Image capture with identification of illuminant

ABSTRACT

Image capture of a scene in which there is an identification of an imaging property of the scene, such as the illuminant or dynamic range of the scene. An imaging assembly has a spectral response which is tunable in accordance with a capture parameter, and first and second different capture parameters are applied to the image sensor. Respective first and second images of the scene are captured. The first and second images of the scene are compared to identify an imaging property of the scene, such as the illuminant or multiple illuminants in respective ones of multiple regions in the scene. In accordance with the property identified for the scene, a third capture parameter is derived, such as to obtain color correction or white balance in accordance with each identified illuminant. The third capture parameter is applied to the image sensor, and a final image of the scene is captured.

FIELD

The present disclosure relates to image capture with an identificationof an imaging property of a scene being captured, such as anidentification of the illuminant or illuminants of the scene beingcaptured, and more particularly relates to both still and video imagecapture with good color balance.

BACKGROUND

In human color vision, color constancy is the ability of the visualsystem to preserve the appearance of an object under a wide range oflight sources. For example, because of color constancy in human colorvision, colored objects are perceived such that they largely maintaintheir color appearance even under illuminants that differ greatly.

Imaging systems, either film or electronic photo-sensors, lack thisability and thus do not exhibit color constancy. It is thereforeincumbent on the operator to ensure that an image is captured with goodcolor balance, for example, white balance.

In shooting film, color balance is typically achieved by using colorcorrection filters over lighting for the scene or over the camera lens.Image data acquired by imaging photo-sensors must be transformed fromthe acquired values to new values that are appropriate for colorreproduction or display. Several aspects of the acquisition and displayprocess make such color correction important, including the facts thatthe acquisition sensors do not match the sensors in the human eye; thatthe properties of the display medium must be accounted for; and that theambient viewing conditions of the acquisition differ from the displayviewing conditions.

In photography and image processing, color balance is sometimes thoughtof as the global adjustment of the intensities of the colors (typicallyred, green, and blue primary colors). An important goal of thisadjustment is to render specific colors, in particular neutral colors,correctly; hence, the general method is sometimes called gray balance,neutral balance, or white balance. Color balance changes the overallmixture of colors in an image and is used for color correction. An imagethat is not color balanced is said to have a color cast, or to exhibitcolor failure, as everything in the image appears to have been shiftedtowards one color or another. Color balancing may be thought in terms ofremoving this color cast.

Algorithms and techniques used to attain color constancy are frequentlyused for color balancing. Conceptually, color balancing consists of twosteps: first, determining the illuminant under which an image wascaptured; and second, scaling the channels of the image to eliminate thecolor cast. There is a large literature on how one might estimate theambient illumination from the camera data and then use this informationto transform the image data. A variety of algorithms have been proposedsuch as Bayesian method, artificial neural network, or retinex. Anexample of a Bayesian method is provided at G. Finlayson, P. M. Hubeland S. Hordley, “Color by correlation: a simple, unifying framework forcolor constancy”, IEEE Transactions on Pattern Analysis and MachineIntelligence 23, 1209-1221 (2001). An example of artificial neuralnetwork is provided at B. Funt, V. Cardei and K. Barnard, “Learningcolor constancy”, in Proceedings of the Fourth IS&T/SID Color ImagingConference, pp. 58-60, 1996. An example of retinex is provided at E. H.Land and J. McCann, “Lightness and retinex theory”, J. Opt. Soc. Am. 61,1-11 (1971).

Currently, the most common implementation of illuminant estimation indigital cameras is based on variations of Bayesian method by calculatingratios of the channels, for example, red/green or red/blue andestimating the illumination based on statistics of these ratios.

SUMMARY

There is no known color balance algorithm that always works accuratelyin conventional cameras for arbitrary scenes with arbitrary illuminants.This limitation is related to issues of robustness, scenes with multipleilluminants with different color temperatures, and irrecoverability of abalanced scene, among others. These issues are explored more fullybelow.

Robustness: Statistical methods are based on training sets andstatistics based on scene assumptions. Moreover, with three colorchannels with fixed sensitivities, there are not sufficient degrees offreedom to unequivocally find out the correct illuminant.

Consider a simple example using a conventional digital camera whichoutputs RGB color signals for each pixel with RGB color spectralsensitivities that mimic the RGB color sensitivities of the human eye.Further consider the use of such a camera in two different scenarios: afirst scenario where the camera captures an image of a white sheet ofpaper under the yellowish illumination of a tungsten-halogen illuminant,and a second scenario where the same camera captures an image of ayellow sheet of paper under the blue-ish illumination of a daylightilluminant such as D65.

Despite the significant differences in these two scenarios, with respectto the data for pixels representing the sheets of paper, the RGBcomponents of these data will be nearly identical. That is, aconventional digital camera is unable to differentiate between a whitesheet of paper under a yellowish tungsten-halogen illuminant, and ayellow sheet of paper under a daylight illuminant. As a consequence,despite the material differences in these scenarios, a conventionaldigital camera is unable to correctly deduce the nature of theilluminant.

Scenes with multiple illuminants with different color temperatures: Evenin scenes with multiples illuminants, conventional illuminant estimationis performed globally. As a consequence, the estimated illuminant fornearly every region of the scene will be wrong, leading to color castsfor all areas covered by every illumination. In order to correct allcolor cast areas, it is necessary to consider a method that allows colorbalance for regions illuminated by each illuminant.

Consider for example a scene having different illuminants in the leftand right halves, such as a scene in whose left half a yellow paper isilluminated by daylight illuminant with blue color cast while in itsright half a white paper is illuminated by halogen lamp with a yellowcast. This type of scene can happen in daily life, for example, a livingroom scene where a reading halogen lamp illuminates a book while thereare also objects in the same scene illuminated by external daylightthrough the window.

Other examples of scenes with multiple illuminants with differentcorrelated color temperatures (CCT) are easily conceived. Consideranother example of a mountain scene at dusk in which the mountain isilluminated with horizon light from sunset with correlated colortemperature (CCT) of around 2,000 K, whereas surrounded valleys areilluminated by a dark blue sky with CCT around 20,000 K. It is evidentthat this scene captured by a conventional RGB camera has color casts inboth regions illuminated directly by sun and indirect illuminant fromdark blue sky. Consider further another example of a typical officescene showing a computer screen with typical CCT of 9,000 K, officefluorescent illumination with CCT of 4,000 K and daylight from exteriorwindows with CCT of around 6,000 K. By setting color balance toautomatic, a conventional digital camera might capture an image whereexterior objects visible through the windows might look balanced, butthe interior of the office and the computer screen both present colorcasts.

Irrecoverability of balanced scene: Images captured with strong castoften cannot be balanced completely because it is common to have asaturated channel or channels. Once a channel is saturated it is notpossible to accurately recover information. Even if the illuminant isestimated correctly it is not possible to recover information that islost. On the other side, unbalanced images often have a low-level noisychannel that will deteriorate the final image quality because ofscaling. Perhaps one technique to avoid this issue is by adjusting thechannel sensitivities before capturing the image or by using theappropriate filtering during capture, although this latter approachdecreases overall sensitivity.

In recognition of the foregoing, the specification herein describesfurther advances in the current state of the art.

Recently, imaging assemblies have been developed in which the imagingassemblies have a tunable spectral response. Two examples of suchimaging assemblies are described here. In the first example of imagingassemblies which have a tunable spectral response, there are imagingassemblies where the image sensor itself has a tunable spectralresponse. For instance, there is an image sensor described in “TheTransverse Field Detector: A Novel Color Sensitive CMOS Device”, Zaraga,IEEE Electron Device Letters 29, 1306-1308 (2008) and U.S. PatentPublication No. 2010/0044822, the contents of which are incorporatedherein by reference. These documents describe a transverse fielddetector (TFD) which has a tunable spectral responsivity that can beadjusted by application of bias voltages to control electrodes. Eachpixel outputs signals for a red-like channel, a green-like channel, anda blue-like channel

In some of these image sensors, the spectral responsivity is tunableglobally, meaning that all pixels in the image sensor are tuned globallyto the same spectral responsivity.

In some others of these image sensors, the spectral responsivity istunable on a pixel by pixel basis or a region-by-region basis. Biasvoltages are applied in a grid-like spatial mask, such that the spectralresponsivity of each pixel is tunable individually of other pixels inthe image sensor, or such that the spectral responsivity of each regioncomprising multiple pixels is tunable individually of other regions inthe image sensor.

In the second example of imaging assemblies which have a tunablespectral response, there are imaging assemblies where the image sensoris preceded by a color filter array (CFA), and it is the color filterarray that has a tunable spectral response. In the first exampledescribed above, because the image sensor itself has a tunable spectralresponse, it might be customary to omit a preceding color filter array,since the inclusion of any filter necessarily would decrease thesignal-to-noise ratio by filtering the amount of light incident on theimage sensor. In contrast, in this second example, the spectralresponsivity of the image sensor is not necessarily tunable, but thespectral responsivity of a preceding color filter array is. Forinstance, there is a tunable color filter array described in U.S. Pat.No. 6,466,961 by Miller, “Methods for Adaptive Spectral, Spatial andTemporal Sensing for Imaging Applications”, the content of which isincorporated herein by reference. This document describes an imagingassembly comprising a color filter array which precedes an image sensorwhose spectral responsivity is constant, but in which the color filterarray itself has a tunable spectral responsivity that can be adjusted byapplication of bias voltages to control electrodes. Each array elementthus filters light incident on corresponding pixels of the image sensor,and the image sensor thereafter outputs signals from which a red-likechannel, a green-like channel, and a blue-like channel, can all bederived for each pixel. In the case of a color filter array withtemporal sensing, the channels for each pixel may be outputsequentially, one after the other. In the case of a color filter arraywith spatial sensing, the channels for each pixel may be outputsimultaneously or nearly so, although demosaicing might be requireddepending on the geometry of the color filter array.

In some of these color filter arrays, the spectral response is tunableglobally, resulting in a situation where corresponding channels for allpixels in the image sensor are tuned globally to the same spectralresponsivity.

In some others of these color filter arrays, the spectral responsivityis tunable on a pixel by pixel basis or a region-by-region basis. Biasvoltages are applied in a grid-like spatial mask, such that the spectralresponsivity for each pixel is tunable individually of other pixels, orsuch that the spectral responsivity for each region comprising multiplepixels is tunable individually of other regions.

According to an aspect of the disclosure herein, an image-capture deviceis provided with an imaging assembly configured to capture an image andacquire image data. The imaging assembly has tunable sensitivitiescontrolled by a capture parameter, such as voltages applied to eachpixel of an image sensor which has tunable spectral responsivity, orsuch as voltages that are applied to each array element of a colorfilter array which has tunable spectral responsivity. The colorsensitivities for each pixel are thereby adjusted in accordance with aspatial mask.

According to another aspect of the disclosure herein, an imagingproperty for a scene is identified, such as a color balance property ora high-dynamic range property. Two images of the scene are captured withtwo sets of different capture parameters. The capture parameters may beapplied, and the two images of the scene captured, when a shutter ishalf pressed. The two sets of capture parameters may be pre-designated,and the differences are designed to provide good discrimination for theimaging property. The imaging property for the scene is identified basedon a comparison of the two images. The imaging property may beidentified individually for different regions of the scene, in whichcase the two captured images may be divided into plural regions based onstatistics of the images. In some embodiments, based on the imagingproperty (or properties) thus identified, a third capture parameter maybe derived, wherein the third capture parameter is designed for captureof the scene with accuracy for the identified imaging property. Thethird capture parameter is applied to a tunable imaging assembly and afinal image of the scene is captured, such as when a shutter is fullypressed.

According to another aspect of the disclosure herein, automatic colorbalance adjustments on an image-capture device is provided. Twoconsecutive scenes are captured with two sets of different captureparameters such as capture parameters that alter spectral sensitivities.The capture parameters may be applied by applying two sets ofpre-programmed spatial masks when a shutter is half pressed. Thepre-programmed spatial masks are designed and optimized to provide gooddiscrimination between individual ones of a wide range of illuminants.The image is divided into a plurality of regions according to statisticsfrom the captured channels. Color sensitivities for each region of thescene are adjusted based on a comparison of the two captured scenes.Each region from the plurality of regions may have the color balancevalue adjusted by providing appropriate voltage spatial mask rather thana global color balance adjustment. Thus, not only is a more robust colorbalance provided, but also color balancing of scenes with multipleilluminants.

According to yet another aspect of the disclosure herein, high-dynamicrange imaging on an image-capturing device is provided. Two consecutivescenes are captured with two sets of different capture parameters suchas capture parameters that alter spectral sensitivities. The captureparameters may be applied by applying two sets of pre-programmed spatialmasks when a shutter is half pressed. The spatial masks are designed tocapture highlight and shadows of the scene. The image is divided into aplurality of regions according to the statistics from the capturedchannels. Color sensitivities for each region of the scene are adjustedbased on a comparison of the two captured scenes. Each region from theplurality of regions may have the exposure value adjusted by providingappropriate voltage spatial mask rather than a global exposureadjustment. Thus, a more efficient way is provided for capture ofhigh-dynamic range scenes in one shot after scene analysis.

According to yet another aspect of the disclosure herein, spectralimaging on an image-capturing device is provided. Two consecutive scenesare captured with two sets of different capture parameters such ascapture parameters that alter spectral sensitivities. The captureparameters may be applied by applying two sets of pre-programmed spatialmasks when a shutter is half pressed. The spatial masks are designed tocapture the diversity of spectral information in the scene. The image isdivided into a plurality of regions according to the statistics from thecaptured channels. Color sensitivities for each region of the scene areadjusted based on a comparison of the two captured scenes, to providecolor sensitivities that correlate with the wavelengths of relevantfeatures from each areas of the scene. Each region from the plurality ofregions may have the exposure value adjusted by providing appropriatevoltage spatial mask. Thus, not only is a more efficient way providedfor capture of spectral images in one shot after scene analysis, butalso customization of color sensitivities to capture important spectralfeatures for each region of the scene.

In yet another aspect, the disclosure herein describes image capture ofa scene in which the illuminant or illuminants of the scene areidentified, such as for use in color correction or white balance. Animage capture device includes an imaging assembly having a spectralresponse which is tunable in accordance with a capture parameter, suchas an imaging assembly with an image sensor which has a tunable spectralresponse or an imaging assembly with an image sensor and a precedingcolor filter array which has a tunable spectral response. A firstcapture parameter is applied to the imaging assembly, and a first imageof the scene is captured using the imaging assembly whose spectralresponse is tuned in accordance with the first capture parameter. Asecond capture parameter is applied to the imaging assembly, wherein thesecond capture parameter is different from the first capture parameter,and a second image of the scene is captured by using the imagingassembly whose spectral response is tuned in accordance with the secondcapture parameter. The first and second images of the scene are comparedto identify an illuminant for the scene, or to identify multipleilluminants in respective ones of multiple regions in the scene. Inaccordance with the illuminant(s) identified for the scene, a thirdcapture parameter is derived wherein the third capture parameter isderived so as to obtain white balance in accordance with each identifiedilluminant. The third capture parameter is applied to the imagingassembly, and a final image of the scene is captured, or an additionaliteration is applied so as to fine-tune color balance of the scene.

In the comparison of the first and second images, the first and secondimages are compared to identify multiple regions of the scene having asimilar illuminant, or to identify multiple regions in the scene havingilluminants that differ from one another. In such an embodiment,multiple third capture parameters may be derived, one each for each ofsuch multiple regions, wherein each such third capture parameter isapplied to the imaging assembly in correspondence to each of suchmultiple regions.

In some embodiments described herein, the first and second captureparameters are sufficiently different from each other so as to providegood discrimination for the imaging property of interest. For example,the first and second capture parameters may differ from each other suchthat first and second image data captured under a variety of differentilluminants differs by more than a threshold value as between eachdifferent pair of illuminants. In some embodiments, one set of captureparameters might cause a blue-shift in spectral sensitivity relative tothe other set of capture parameters.

The imaging assembly may provide three or more channels of informationfor each pixel of the scene, including a red-like channel, a blue-likechannel and a green-like channel. In such an embodiment, in thecomparison of the first and second images of the scene, the first andsecond images may be compared by using ratios of the channels.

According to further aspects described herein, an image capture deviceincludes an imaging assembly with a tunable spectral responsivity andhaving an image sensor arranged in an array of pixels. The spectralresponsivity for each pixel is tunable based on a capture parameter suchas a capture parameter in an arrayed mask of capture parameters arrangedin correspondence to the array of pixels. A mask of first captureparameters is applied to the imaging assembly. A first image of a sceneis captured, wherein the first image of the scene is captured by usingthe imaging assembly with spectral responses adjusted in accordance withthe first capture parameters in the mask. Scene properties of the firstimage are analyzed so as to determine one or more imaging properties ofthe scene, such as imaging properties relating to visibility of detailsin the first image of the scene, dynamic range of the first image of thescene, and/or color balance of the first image of the scene. The firstcapture parameters in the mask are modified so as to obtain secondcapture parameters, wherein each capture parameter is modified bycalculations that use the scene parameters, and wherein the calculationsfor one capture parameter are independent of the calculations for allother capture parameters. The mask of second capture parameters isapplied to the imaging assembly, and a second image of the scene iscaptured, wherein the second image of the scene is captured by using theimaging assembly with spectral responses adjusted in accordance with thesecond capture parameters in the mask.

In some embodiments described herein, the first capture parameters areall the same initial and pre-designated capture parameters. Thepre-designated capture parameters may be predesignated pixel-basedspatial values set to compensate for spatial and spectralnon-uniformities between the pixels in the image sensor. The sceneparameters may indicate an amount of compensation needed for colorbalance, and/or for signal attenuation or boost.

In some embodiments, there may be an iterated repetition of an analysisof scene properties of the second image so as to determine one or moreof the aforementioned imaging properties of the scene; a modification ofthe second capture parameters in the mask so as to obtain third captureparameters, wherein each capture parameter is modified by calculationsthat use the scene parameters, and wherein the calculations for onecapture parameter are independent of the calculations for all othercapture parameters; an application of the mask of third captureparameters to the imaging assembly; and a capture of a third image ofthe scene, wherein the third image of the scene is captured by using theimaging assembly with spectral responses adjusted in accordance with thethird capture parameters in the mask. The scene parameters may includean iteration flag indicative of whether further iterations are needed,and the scene parameters may include a multi-capture flag indicative ofwhether the scene comprises a high dynamic range (HDR) scene or amultispectral scene. Multiple second images may be captured inaccordance with the multi-capture flag.

In further aspects described herein, there is an iterative correction ofspatial non-uniformity of an image sensor arranged in an array of pixelswith adjustable spectral responsivities for each pixel. Initial captureparameters are applied for each pixel using a pixel-based spatialelectronic mask, and a first image is captured with the initial captureparameters in the pixel-based spatial electronic mask. There is ananalysis of spatial uniformity in light sensitivity and colortemperature of the captured image, followed by a determination ofwhether the spatial uniformity is or is not within pre-establishedtolerances. The pixel-based electronic spatial mask is stored in memoryfor use with further image captures, in a case where the spatialuniformity is determined to be within the pre-established tolerances. Onthe other hand, in a case where the spatial uniformity is determined notto be within the pre-established tolerances, modified capture parametersare generated for each pixel in the pixel-based spatial electronic mask,wherein the modified capture parameters at least partially compensatefor the spatial non-uniformity, and the steps of capturing, analyzingand determining, are repeatedly iterated.

This brief summary has been provided so that the nature of thisdisclosure may be understood quickly. A more complete understanding canbe obtained by reference to the following detailed description and tothe attached drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram showing an example embodiment of a digitalcamera.

FIG. 1A is a view for explaining the architecture of modules accordingto an example embodiment.

FIGS. 2 and 3 are views showing external appearance of an exampleembodiment.

FIG. 4 is a flow diagram for explaining an example embodiment ofautomatic determination of an imaging property for a scene.

FIG. 5 is a view for explaining an example embodiment for selection ofan initial state for an electronic mask which is applied to an imagingsensor in an example embodiment.

FIG. 6 is a view for explaining sheen property analysis and processingfor multiple captured images.

FIG. 7 is a view for explaining an example of scene property analysisfor a high-dynamic range imaging mode.

FIG. 8 is an example of a capture parameter for capturing a first image.

FIGS. 9 and 10 are views showing spectral radiance of a tungsten-halogenilluminant and a daylight illuminant, respectively.

FIGS. 11 and 12 are views showing spectral reflectance of white paperand yellow paper, respectively.

FIG. 13 is an example of a second capture parameter for capturing asecond image.

FIG. 14 is a view showing a third capture parameter derived inaccordance with an illuminant identified for a scene.

FIGS. 15 and 16 are views for explaining numerical examples whichdifferentiate between illuminants and between regions having differentilluminants.

FIG. 17 is a block diagram showing an example embodiment of a digitalcamera.

FIG. 18 is a flow diagram for explaining an example embodiment ofautomatic determination of an imaging property for a scene.

DETAILED DESCRIPTION

In the following example embodiments, there is described a digitalcamera which may be a digital still camera or a digital video camera. Itis understood, however, that the following description encompassesarbitrary arrangements which can incorporate or utilize such imagingassemblies having a spectral response which is tunable in accordancewith a capture parameter, for instance, a data processing apparatushaving an image sensing function (e.g., a personal computer) or aportable terminal having an image sensing function (e.g., a mobiletelephone).

<FIGS. 1 to 16>

FIG. 1 is a block diagram showing an example of the arrangement of thedigital camera 100 as an image capturing device according to thisembodiment. Referring to FIG. 1, reference numeral 10 denotes an imaginglens; 12, a shutter having an aperture function; and 14, an image sensorwhich has a spectral response which is tunable in accordance with acapture parameter, which converts an optical image into an electricalsignal. Reference numeral 16 denotes an A/D converter which converts ananalog signal into a digital signal. The A/D converter 16 is used whenan analog signal output from the image sensor 14 is converted into adigital signal and when an analog signal output from an audio controller11 is converted into a digital signal. Reference numeral 102 denotes ashield, or barrier, which covers the image sensor including the lens 10of the digital camera 100 to prevent an image capturing system includingthe lens 10, shutter 12, and image sensor 14 from being contaminated ordamaged.

In FIG. 1, an imaging assembly is comprised of image sensor 14 andassociated optics, such that in some embodiments the imaging assembly iscomprised of image sensor 14 and lens 10.

The optical system 10 may be of a zoom lens, thereby providing anoptical zoom function. The optical zoom function is realized by drivinga magnification-variable lens of the optical system 10 using a drivingmechanism of the optical system 10 or a driving mechanism provided onthe main unit of the digital camera 100.

A light beam (light beam incident upon the angle of view of the lens)from an object in a scene that goes through the optical system (imagesensing lens) 10 passes through an opening of a shutter 12 having adiaphragm function, and forms an optical image of the object on theimage sensing surface of the image sensor 14. The image sensor 14converts the optical image to analog image signals and outputs thesignals to an A/D converter 16. The A/D converter 16 converts the analogimage signals to digital image signals (image data). The image sensor 14and the A/D converter 16 are controlled by clock signals and controlsignals provided by a timing generator 18. The timing generator 18 iscontrolled by a memory controller 22 and a system controller 50.

Image sensor 14 is an image sensor which has a spectral response whichis tunable in accordance with a capture parameter 17. For each pixel,image sensor 14 outputs three or more channels of color information,including a red-like channel, a green-like channel and a blue-likechannel. The precise nature of the spectral responsivity of image sensor14 is controlled via capture parameter 17. In this embodiment, captureparameter 17 may be comprised of multiple spatial masks, with one maskeach for each channel of information output by image sensor 14. Thus, inthis example, where image sensor 14 outputs three or more channels,capture parameter 17 includes a spatial mask DR for the red-like channelof information, a spatial mask DG for the green-like channel ofinformation, and a spatial mask DB for the blue-like channel ofinformation. Each spatial mask comprises an array of control parameterscorresponding to pixels or regions of pixels in image sensor 14. Thespectral responsivity of each pixel, or each region of plural pixels, isthus tunable individually and independently of other pixels or regionsof pixels.

Image sensor 14 may be comprised of a transverse field detector (TFD)sensor mentioned hereinabove. Spatial masks DR, DG and DB may correspondto voltage biases applied to control electrodes of the TFD sensor.

Reference numeral 18 denotes a timing generator, which supplies clocksignals and control signals to the image sensor 14, the audio controller11, the A/D converter 16, and a D/A converter 26. The timing generator18 is controlled by a memory controller 22 and system controller 50.Reference numeral 20 denotes an image processor, which applies resizeprocessing such as predetermined interpolation and reduction, and colorconversion processing to data from the A/D converter 16 or that from thememory controller 22. The image processor 20 executes predeterminedarithmetic processing using the captured image data, and the systemcontroller 50 executes exposure control and ranging control based on theobtained arithmetic result.

As a result, TTL (through-the-lens) AF (auto focus) processing, AE (autoexposure) processing, and EF (flash pre-emission) processing areexecuted. The image processor 20 further executes predeterminedarithmetic processing using the captured image data, and also executesTTL AWB (auto white balance) processing based on the obtained arithmeticresult. It is understood that in other embodiments, optical finder 104may be used in combination with the TTL arrangement, or in substitutiontherefor.

Output data from the A/D converter 16 is written in a memory 30 via theimage processor 20 and memory controller 22 or directly via the memorycontroller 22. The memory 30 stores image data which is captured by theimage sensor 14 and is converted into digital data by the A/D converter16, and image data to be displayed on an image display unit 28. Theimage display unit 28 may be a liquid crystal screen. Note that thememory 30 is also used to store audio data recorded via a microphone 13,still images, movies, and file headers upon forming image files.Therefore, the memory 30 has a storage capacity large enough to store apredetermined number of still image data, and movie data and audio datafor a predetermined period of time.

A compression/decompression unit 32 compresses or decompresses imagedata by adaptive discrete cosine transform (ADCT) or the like. Thecompression/decompression unit 32 loads captured image data stored inthe memory 30 in response to pressing of the shutter 310 as a trigger,executes the compression processing, and writes the processed data inthe memory 30. Also, the compression/decompression unit 32 appliesdecompression processing to compressed image data loaded from adetachable recording unit 202 or 212, as described below, and writes theprocessed data in the memory 30. Likewise, image data written in thememory 30 by the compression/decompression unit 32 is converted into afile by the system controller 50, and that file is recorded in therecording unit 202 or 212, as also described below.

The memory 30 also serves as an image display memory (video memory).Reference numeral 26 denotes a D/A converter, which converts imagedisplay data stored in the memory 30 into an analog signal, and suppliesthat analog signal to the image display unit 28. Reference numeral 28denotes an image display unit, which makes display according to theanalog signal from the D/A converter 26 on the liquid crystal screen 28of an LCD display. In this manner, image data to be displayed written inthe memory 30 is displayed by the image display unit 28 via the D/Aconverter 26.

The exposure controller 40 controls the shutter 12 having a diaphragmfunction based on the data supplied from the system controller 50. Theexposure controller 40 may also have a flash exposure compensationfunction by linking up with a flash (flash emission device) 48. Theflash 48 has an AF auxiliary light projection function and a flashexposure compensation function.

The distance measurement controller 42 controls a focusing lens of theoptical system 10 based on the data supplied from the system controller50. A zoom controller 44 controls zooming of the optical system 10. Ashield controller 46 controls the operation of a shield (barrier) 102 toprotect the optical system 10.

Reference numeral 13 denotes a microphone. An audio signal output fromthe microphone 13 is supplied to the A/D converter 16 via the audiocontroller 11 which includes an amplifier and the like, is convertedinto a digital signal by the A/D converter 16, and is then stored in thememory 30 by the memory controller 22. On the other hand, audio data isloaded from the memory 30, and is converted into an analog signal by theD/A converter 26. The audio controller 11 drives a speaker 15 accordingto this analog signal, thus outputting a sound.

A nonvolatile memory 56 is an electrically erasable and recordablememory, and uses, for example, an EEPROM. The nonvolatile memory 56stores constants, computer-executable programs, and the like foroperation of system controller 50. Note that the programs include thosefor execution of various flowcharts.

In particular, and as shown in FIG. 1A, non-volatile memory 56 is anexample of a non-transitory computer-readable memory medium, havingstored thereon camera control modules 74 as described herein. Alsostored thereon are pre-designated capture parameters for application toimage sensor 14 so as to control spectral responsivity of the imagesensor. In this embodiment, the capture parameters are comprised ofspatial masks 75 so as to permit pixel-by-pixel or region-by-regioncontrol of spectral responsivity, independently of other pixels orregions. A spatial mask generator 76 generates masks, such as byproviding one of pre-designated masks 75 or by deriving a new mask. Thederived mask may be based on a comparison of captured images, or may bebased on a comparison of scene properties as provided by scene propertyanalysis module 77.

Reference numeral 50 denotes a system controller, which controls theentire digital camera 100. The system controller 50 executes programsrecorded in the aforementioned nonvolatile memory 56 to implementrespective processes to be described later of this embodiment. Referencenumeral 52 denotes a system memory which comprises a RAM. On the systemmemory 52, constants and variables required to operate system controller50, programs read out from the nonvolatile memory 56, and the like aremapped.

A mode selection switch 60, shutter switch 310, and operation unit 70form operation means used to input various operation instructions to thesystem controller 50.

The mode selection switch 60 includes the imaging/playback selectionswitch, and is used to switch the operation mode of the systemcontroller 50 to one of a still image recording mode, movie recordingmode, playback mode, and the like.

The shutter switch 62 is turned on in the middle of operation (halfstroke) of the shutter button 310 arranged on the digital camera 100,and generates a first shutter switch signal SW1. Also, the shutterswitch 64 is turned on upon completion of operation (full stroke) of theshutter button 310, and generates a second shutter switch signal SW2.The system controller 50 starts the operations of the AF (auto focus)processing, AE (auto exposure) processing, AWB (auto white balance)processing, EF (flash pre-emission) processing, and the like in responseto the first shutter switch signal SW1. Also, in response to the secondshutter switch signal SW2, the system controller 50 starts a series ofprocessing (shooting) including the following: processing to read imagesignals from the image sensing device 14, convert the image signals intoimage data by the A/D converter 16, process the image data by the imageprocessor 20, and write the data in the memory 30 through the memorycontroller 22; and processing to read the image data from the memory 30,compress the image data by the compression/decompression circuit 32, andwrite the compressed image data in the recording medium 200 or 210.

A zoom operation unit 65 is an operation unit operated by a user forchanging the angle of view (zooming magnification or shootingmagnification). The operation unit 65 can be configured with, e.g., aslide-type or lever-type operation member, and a switch or a sensor fordetecting the operation of the member.

The image display ON/OFF switch 66 sets ON/OFF of the image display unit28. In shooting an image with the optical finder 104, the display of theimage display unit 28 configured with a TFT, an LCD or the like may beturned off to cut the power supply for the purpose of power saving.

The flash setting button 68 sets and changes the flash operation mode.In this embodiment, the settable modes include: auto, flash-on, red-eyereduction auto, and flash-on (red-eye reduction). In the auto mode,flash is automatically emitted in accordance with the lightness of anobject. In the flash-on mode, flash is always emitted whenever shootingis performed. In the red-eye reduction auto mode, flash is automaticallyemitted in accordance with lightness of an object, and in case of flashemission the red-eye reduction lamp is always emitted whenever shootingis performed. In the flash-on (red-eye reduction) mode, the red-eyereduction lamp and flash are always emitted.

The operation unit 70 comprises various buttons, touch panels and so on.More specifically, the operation unit 70 includes a menu button, a setbutton, a macro selection button, a multi-image reproduction/repagingbutton, a single-shot/serial shot/self-timer selection button, a forward(+) menu selection button, a backward (−) menu selection button, and thelike. Furthermore, the operation unit 70 may include a forward (+)reproduction image search button, a backward (−) reproduction imagesearch button, an image shooting quality selection button, an exposurecompensation button, a date/time set button, a compression mode switchand the like.

The compression mode switch is provided for setting or selecting acompression rate in JPEG (Joint Photographic Expert Group) compression,recording in a RAW mode and the like. In the RAW mode, analog imagesignals outputted by the image sensing device are digitalized (RAW data)as it is and recorded.

Note in the present embodiment, RAW data includes not only the dataobtained by performing A/D conversion on the photoelectrically converteddata from the image sensing device, but also the data obtained byperforming lossless compression on A/D converted data. Moreover, RAWdata indicates data maintaining output information from the imagesensing device without a loss. For instance, RAW data is A/D convertedanalog image signals which have not been subjected to white balanceprocessing, color separation processing for separating luminance signalsfrom color signals, or color interpolation processing. Furthermore, RAWdata is not limited to digitalized data, but may be of analog imagesignals obtained from the image sensing device.

According to the present embodiment, the JPEG compression mode includes,e.g., a normal mode and a fine mode. A user of the digital camera 100can select the normal mode in a case of placing a high value on the datasize of a shot image, and can select the fine mode in a case of placinga high value on the quality of a shot image.

In the JPEG compression mode, the compression/decompression circuit 32reads image data written in the memory 30 to perform compression at aset compression rate, and records the compressed data in, e.g., therecording medium 200.

In the RAW mode, analog image signals are read in units of line inaccordance with the pixel arrangement of the color filter of the imagesensing device 14, and image data written in the memory 30 through theA/D converter 16 and the memory controller 22 is recorded in therecording medium 200 or 210.

Note that the digital camera 100 according to the present embodiment hasa plural-image shooting mode, where plural image data can be recorded inresponse to a single shooting instruction by a user. Image datarecording in this mode includes image data recording typified by an autobracket mode, where shooting parameters such as white balance andexposure are changed step by step. It also includes recording of imagedata having different post-shooting image processing contents, forinstance, recording of plural image data having different data formssuch as recording in a JPEG form or a RAW form, recording of image datahaving the same form but different compression rates, and recording ofimage data on which predetermined image processing has been performedand has not been performed.

A power controller 80 comprises a power detection circuit, a DC-DCconverter, a switch circuit to select the block to be energized, and thelike. The power controller 80 detects the existence/absence of a powersource, the type of the power source, and a remaining battery powerlevel, controls the DC-DC converter based on the results of detectionand an instruction from the system controller 50, and supplies anecessary voltage to the respective blocks for a necessary period. Apower source 86 is a primary battery such as an alkaline battery or alithium battery, a secondary battery such as an NiCd battery, an NiMHbattery or an Li battery, an AC adapter, or the like. The main unit ofthe digital camera 100 and the power source 86 are connected byconnectors 82 and 84 respectively comprised therein.

The recording media 200 and 210 comprise: recording units 202 and 212that are configured with semiconductor memories, magnetic disks and thelike, interfaces 203 and 213 for communication with the digital camera100, and connectors 206 and 216. The recording media 200 and 210 areconnected to the digital camera 100 through connectors 206 and 216 ofthe media and connectors 92 and 96 of the digital camera 100. To theconnectors 92 and 96, interfaces 90 and 94 are connected. Theattached/detached state of the recording media 200 and 210 is detectedby a recording medium attached/detached state detector 98.

Note that although the digital camera 100 according to the presentembodiment comprises two systems of interfaces and connectors forconnecting the recording media, a single or plural arbitrary numbers ofinterfaces and connectors may be provided for connecting a recordingmedium. Further, interfaces and connectors pursuant to differentstandards may be provided for each system.

For the interfaces 90 and 94 as well as the connectors 92 and 96, cardsin conformity with a standard, e.g., PCMCIA cards, compact flash (CF)(registered trademark) cards and the like, may be used. In this case,connection utilizing various communication cards can realize mutualtransfer/reception of image data and control data attached to the imagedata between the digital camera and other peripheral devices such ascomputers and printers. The communication cards include, for instance, aLAN card, a modem card, a USB card, an IEEE 1394 card, a P1284 card, anSCSI card, and a communication card for PHS or the like.

The optical finder 104 is configured with, e.g., a TTL finder, whichforms an image from the light beam that has gone through the lens 10utilizing prisms and minors. By utilizing the optical finder 104, it ispossible to shoot an image without utilizing an electronic view finderfunction of the image display unit 28. The optical finder 104 includesindicators, which constitute part of the display device 54, forindicating, e.g., a focus state, a camera shake warning, a flash chargestate, a shutter speed, an f-stop value, and exposure compensation.

A communication circuit 110 provides various communication functionssuch as USB, IEEE 1394, P1284, SCSI, modem, LAN, RS232C, and wirelesscommunication. To the communication circuit 110, a connector 112 can beconnected for connecting the digital camera 100 to other devices, or anantenna can be provided for wireless communication.

A real-time clock (RTC, not shown) may be provided to measure date andtime. The RTC holds an internal power supply unit independently of thepower supply controller 80, and continues time measurement even when thepower supply unit 86 is OFF. The system controller 50 sets a systemtimer using a date and time obtained from the RTC at the time ofactivation, and executes timer control.

FIGS. 2 and 3 are views showing an example of an external appearance ofthe digital camera 100. Note in these figures, some components areomitted for description purpose. The aforementioned operation unit 70comprises, e.g., buttons and switches 301 to 311. A user operates thesebuttons and switches 301 to 311 for turning ON/OFF the power of thedigital camera 100, for setting, changing or confirming the shootingparameters, for confirming the status of the camera, and for confirmingshot images.

The power button 311 is provided to start or stop the digital camera100, or to turn ON/OFF the main power of the digital camera 100. Themenu button 302 is provided to display the setting menu such as shootingparameters and operation modes of the digital camera 100, and to displaythe status of the digital camera 100. The menu has, e.g., a hierarchicalstructure, and each hierarchy includes selectable items or items whosevalues are variable.

A delete button 301 is pressed for deleting an image displayed on aplayback mode or a shot-image confirmation screen. In the presentembodiment, the shot-image confirmation screen (a so-called quick reviewscreen) is provided to display a shot image on the image display unit 28immediately after shooting for confirming the shot result. Furthermore,the present embodiment is constructed in a way that the shot-imageconfirmation screen is displayed as long as a user keeps pressing theshutter button 310 after the user instructs shooting by shutter buttondepression.

An enter button 303 is pressed for selecting a mode or an item. When theenter button 303 is pressed, the system controller 50 sets the mode oritem selected at this time. The display ON/OFF button 66 is used forselecting displaying or non-displaying of photograph informationregarding the shot image, and for switching the image display unit 28 tobe functioned as an electronic view finder.

A left button 305, a right button 306, an up button 307, and a downbutton 308 may be used for the following purposes, for instance,changing an option (e.g., items, images) selected from plural options,changing an index position that specifies a selected option, andincreasing or decreasing numeric values (e.g., correction value, dateand time).

Half-stroke of the shutter button 310 instructs the system controller 50to start, for instance, AF processing, AE processing, AWB processing, EFprocessing or the like. Full-stroke of the shutter button 310 instructsthe system controller 50 to perform shooting.

The zoom operation unit 65 is operated by a user for changing the angleof view (zooming magnification or shooting magnification) as mentionedabove.

A recording/playback selection switch 312 is used for switching arecording mode to a playback mode, or switching a playback mode to arecording mode. Note, in place of the above-described operation system,a dial switch may be adopted or other operation systems may be adopted.

FIG. 4 is a flow diagram for explaining an example embodiment of anautomatic determination of optimal imaging properties for specificregions of the image and performing adjustments of, for example, lightsensitivity and spectral selectivity on a pixel-by-pixel basis, or aregion-by-region basis, for imaging sensors with tunable spectralproperties.

Briefly, according to FIG. 4, an imaging property for a scene isidentified, such as a color balance property or a high-dynamic rangeproperty. Two images of the scene are captured with two sets ofdifferent capture parameters. The capture parameters may be applied, andthe two images of the scene captured, when a shutter is half pressed.The two sets of capture parameters may be pre-designated, and thedifferences between them are designed to provide good discrimination forthe imaging property. The imaging property for the scene is identifiedbased on a comparison of the two images. The imaging property may beidentified individually for different regions of the scene, in whichcase the two captured images may be divided into plural regions based onstatistics of the images. In some embodiments, based on the imagingproperty (or properties) thus identified, a third capture parameter maybe derived, wherein the third capture parameter is designed for captureof the scene with accuracy for the identified imaging property. Thethird capture parameter is applied to the imaging assembly and a finalimage of the scene is captured, such as when a shutter is fully pressed.

In step S401, an imaging controller controls spatial electronic maskgenerator 76 to set-up an initial state for a pixel-by-pixel basisspatial electronic voltage mask that is going to modulate the amplitudeand spectral selectivity of an imaging sensor with tunable colorsensitivities. The electronic mask can control amplitude and spectratuning for each pixel. The initial state for the pixel-by-pixel basisspatial mask is given by electronic voltages that has some assumptionsabout illumination and material properties of the scene and is usually apre-designated setting determined in advance such as by a calibrationprocedure that is made in the imaging system assembly line.

One possible example for selection of an initial state for theelectronic mask is shown in FIG. 5, which shows an adaptive method todetermine initial state pixel-based spatial electronic voltage mask. Inthis example, one possible setting is by adjusting the voltage in theinitial state to produce uniform neutral response for a perfectlyuniform and diffuse grey card under D50 illumination. Note that inactual imaging sensors there are non-uniformities in the response ofindividual pixels due to manufacturing tolerances and the optics usedwith the sensor will further produce non-uniformities in color andsensitivity. Therefore the voltage values generated for thepixel-by-pixel basis spatial electronic mask are not the same for allpixels, but they ordinarily have values that produce the same image dataunder the calibration conditions described above. By providing a systemfor pixel-by-pixel calibration of a tunable imaging sensor it ispossible to: (a) compensate for non-uniformities in sensitivity andspectral response in the sensor due to manufacturing; and (b) compensatefor non-uniformities in sensitivity and spectral response due to opticalaberrations and distortions.

As shown in FIG. 5, all values of the pixel-based spatial electronicvoltage mask are set to same default factory value. In step S501, theimaging controller controls the imaging sensor with tunable filters toan initial state mask and captures an image for calibration (steps S502and S503). In step S504, spatial uniformity analysis is performed and ifin step S505 the captured image spatial uniformity is sufficientaccording to a pre-determined spatial uniformity tolerance, then in stepS506 the pixel-based spatial electronic voltage masks is saved in thememory.

If the spatial uniformity of the captured image for calibration is notwithin specified tolerance, then in step S508 a compensation value iscalculated for each pixel and sent to the pixel-based spatial electronicmask generator that creates a new pixel-based spatial electronic voltagemask. Then, in an iterated repetition of step S501, the imagingcontroller then sends command to the imaging sensor with tunable filtersto captures a new calibration image and the captured image forcalibration is analyzed for spatial uniformity. This iterative processis repeated until spatial uniformity of the captured image is within thespecified tolerance.

The electronic mask for the initial state can be stored in a memory unitonce the imaging system is calibrated and it is used every time theimaging system is turned on. The calibration procedure can be repeatedfor different lenses and illuminants and the calibration saved in thememory unit.

This adaptive imaging system is very versatile and can be employed inseveral pre-programmed modes. Each mode can be automatically detected bythe imaging controller as well. For illustration, some frequently usefulmodes are: single image enhancement, high-dynamic range imaging andmulti-spectral imaging, but the adaptive imaging system is not limitedto these applications.

In the example of FIG. 4, a single image enhancement mode is shown.Returning to FIG. 4, in response to the imaging controller, thepixel-based spatial electronic mask generator supplies a spatial mask(step S402), which in step S403 is applied to the imaging sensor. Insteps S404 and S405, the image controller sends a command to the imagingsensor with tunable filters to capture an image using a pre-determinedpixel-based spatial electronic mask that was calculated in thecalibration stage.

The captured image is analyzed by scene property analysis module 77 instep S406, to determine detail visibility and color balance. Forexample, shadows in the scene might be too dark to see any detail andhighlights of the scene might be saturated. In another example, a scenemight be taken under multiple illumination sources with different colortemperature, such that the color balance has to be corrected fordifferent parts of the image. The scene property analysis moduleproduces scene parameters (step S407) that can be used to determine ifmore analysis of scenes is needed (step S408) and which are alsoprovided to the electronic mask generator for determination of a revisedspatial mask. Such scene parameters can include, but not limited to,amount of compensation necessary for color balance, signal attenuationor boosting for each area of the image with corresponding imagecoordinates.

The scene parameters also include a flag signal that indicates if moreanalysis of scenes is needed, meaning that there are more corrections tobe performed and further iterations of scene capture is warranted. Ifmore analysis is necessary the pixel-based spatial electronic maskgenerator generates a new pixel-based spatial electronic voltage maskswith the scene parameters and the image controller controls the imagingsensor with tunable filters to capture a new image. This iterativeprocess continues until a satisfactory image is produced as a finalimage (steps S410 and S411). This scene property analysis module alsodetermines if a mode with multiple image capture (for high-dynamic rangeor multispectral imaging) modes are necessary and switches mode forthese more advanced modes. There are scenes that can be properlycompensated pixel-by-pixel basis in terms of light sensitivity and colorbut there are scenes whose physical values go beyond what can becaptured by one single capture. In such cases, there may be the need tocapture multiple images.

FIG. 6 shows processing for multiple captured images. In particular, insome the high-dynamic range and multispectral imaging modes, the imagingcontroller sends commands for generation of multiple pixel-based spatialelectronic voltage masks for each capture using the imaging sensor withtunable filters to generate multiple captured images, which are shown inFIG. 6. Each captured image is analyzed by the scene property analysismodule and multiple scene parameters are generated. The signalparameters have same function as described above for the single imageenhancement mode with an additional function to provide the final imagerendering module with information of how many images have to be combinedand how they are going to be combined to produce the final image.

Thus, the present disclosure contemplates apparatus and method foradaptive imaging to automatically determine optimal imaging parametersfor specific regions of the image and perform adjustments of lightsensitivity and spectral selectivity on a pixel-by-pixel basis forimaging sensors with tunable spectral properties. As seen herein, thereare in combination an imaging sensor with tunable spectralresponsivities, an imaging controller that controls the imaging sensorand also a spatial pixel-by-pixel basis electronic mask generation unitthat generates pixel-based spatial electronic masks to control the shapeof the sensitivity curves of the imaging sensor with tunable spectralresponsivities, a scene property analysis module that analyzes theproperties of scene based on images captured by the imaging sensor withtunable spectral responsivities using electronic control signalsgenerated by the electronic mask generation unit, and a module thatdecides to generate new spatial electronic control masks from theresults of the scene analysis or to render the final image. One suchimaging sensor may be a transverse field detector (TFD) sensor, and theimaging sensor may capture multiple images. The scene propertiesanalyzed may include dynamic range and spectral properties of the scene(objects and illumination).

Also contemplated herein are iterative methods and apparatus for spatialnon-uniformity correction based on an imaging sensor with tunablespectral responsivities, comprising an imaging sensor with tunablespectral responsivities, an initial state for the pixel-based spatialelectronic mask, an imaging control that captures an image with initialstate pixel-based spatial electronic mask, a scene analysis module thatanalysis spatial uniformity of captured image, and a decision modulethat decides if the spatial uniformity is within pre-establishedtolerances or not. If the criteria of spatial uniformity are met thefinal pixel-based electronic spatial mask is saved in a memory unit. Ifthe criteria are not met the method goes to the next iteration bygenerating appropriate spatial compensation for the spatialnon-uniformity in light sensitivity and color and appropriatepixel-basis spatial electronic masks are generated for the subsequentimage capture.

The scene property analysis of this example works in the followingiterative fashion. First, exposure is adjusted to eliminate or reduce orminimize saturated information in the captured image. Once an imagetaken with correct exposure (leading to minimum saturation) is taken,the scene is analyzed to ensure that there is adequate detail such thatacceptable details are visible. If all the details are visible there isno need to take additional pictures. If exposure time is too short forcertain areas of the image, the sensitivities of the pixelscorresponding to the darker areas of the image have to be boosted (bychanging gain electronically in the imaging sensor pixels) beforeanother picture is taken. The picture is then analyzed again and ifthere is still need to take more pictures with increased sensitivitiesand so on until all or an acceptable amount of details can be captured.In the implementation of the system it is possible to consider astopping condition (for example, maximum number of captures orlimitations in the range of the tunable filters in the imaging sensor).

For illustrative purposes, five (5) parameters are described, but thedisclosure is not limited only to those parameters. Parameter 1 is acontinuity flag that indicates to the pixel-based spatial electronicmask generator if there is a need to capture a new image (the pixel-basespatial electronic mask generator communicates this to the imagingcontroller). Parameter 2 indicates the exposure value (that could beconverted to exposure time, aperture or a combination of both).Parameter 3 is a flag that indicates if it is necessary to change thespatial mask. Parameter 4 has the mask coordinates for gain adjustments.These image coordinates are sent to the pixel-based spatial electronicmask generator to indicate which gains will be applied to which pixels(given by the coordinates), thereby generating an electronic maskaccordingly. Parameter 5 contains the color mask coordinates indicatingwhich color bias has to be applied to which pixel coordinates to thepixel-based spatial electronic mask generator to generate appropriateelectronic masks. If parameter 3 is set to no change, there is no needfor parameters 4 and 5.

FIG. 7 is a view for explaining an example of scene property analysisfor a high-dynamic range imaging mode. The scene property analysis isdepicted in FIG. 7 for a high-dynamic range imaging mode. In this modethe scene property analysis module receives an image at S701 (such asCaptured Image 1 of FIG. 6) and exposure corresponding to the image.

An image histogram generation module at S702 produces a luminancechannel histogram at S703 that is analyzed at S704 by a luminancehistogram analysis module to produce analysis results at S705. Typicallya histogram analysis will look for percentiles to determine if an imageis appropriately exposed but it is not limited to this type of analysis.Based on the analysis results, S706 determines if the image issaturated. One reason for this step is the implicit assumption that mostof the image is not saturated. If the image is saturated it may benecessary to decrease the exposure and a new exposure is estimated atS707 based on analysis results and current exposure. If the image issaturated a new exposure is generated to unsaturate the image andparameter 1 is set to continue the iteration, parameter 2 is set withnew exposure and parameter 3 is set to produce no mask change. Thisprocess is repeated until the image is not saturated.

After determining that saturation is acceptable, S708 tests whetherthere is an acceptable amount of detail, meaning dark areas in the sceneare properly exposed, based on the histogram analysis results. If alldetails are visible parameter 1 is set to stop the iteration (and thisresult will render the image), parameter 2 is set with current exposureand parameter 3 is set to not produce any mask change. However, ifdetails are not visible there is a need to generate new luminance maskcoordinates. At S709, new luminance mask coordinates are generated basedon analysis results and the image. The analysis results give thethreshold of luminance value to be used to select pixels that has to becaptured with new exposure values. Parameter 1 is set to continueiteration. Parameter 2 is current exposure. Parameter 3 indicates thatthere is a new electronic mask and parameter 4 contains the gain biasand coordinates for the mask generation.

In the embodiment of FIG. 7, multi-illumination compensation in a sceneis also addressed by analyzing color channel histograms generated by theimage histogram generation module. The color histograms are analyzed asindicated generally at S710 to estimate illuminant (S711) per regionthat is used in conjunction with image to generate color maskcoordinates. As indicated at S712, color mask coordinates are generated,and contain information on color bias and pixel coordinates for storagein parameter 5. This process is repeated until the unsaturated imageproduces an image that has good visibility. The color processing moduleshould guarantee that the final rendering image is well balanced.

The same scene property analysis mode can be used in a multi-spectralimaging system by storing images with different color bias settings andrendering image in the end of the process.

Other examples may be developed in accordance with the descriptionherein for use of an imaging assembly which has a spectral responsewhich is tunable in accordance with a capture parameter, such as animaging assembly with an image sensor which has a tunable spectralresponse or an imaging assembly with an image sensor and a precedingcolor filter array which has a tunable spectral response. Such examplesmay address color balancing, dynamic range adjustment, spectral imagingindependently for several areas of the imaging frame with one captureafter image analysis, and/or combinations thereof. For purposes ofillustration, the following description focuses almost entirely on colorbalance.

In accordance with this aspect, the disclosure herein describes imagecapture of a scene in which the illuminant or illuminants of the sceneare identified, such as for use in color correction or white balance. Animage capture device includes an imaging assembly having a spectralresponse which is tunable in accordance with a capture parameter, suchas an imaging assembly with an image sensor which has a tunable spectralresponse or an imaging assembly with an image sensor and a precedingcolor filter array which has a tunable spectral response. A firstcapture parameter is applied to the imaging assembly, and a first imageof the scene is captured using the imaging assembly whose spectralresponse is tuned in accordance with the first capture parameter. Asecond capture parameter is applied to the imaging assembly, wherein thesecond capture parameter is different from the first capture parameter,and a second image of the scene is captured by using the imagingassembly whose spectral response is tuned in accordance with the secondcapture parameter. The first and second images of the scene are comparedto identify an illuminant for the scene, or to identify multipleilluminants in respective ones of multiple regions in the scene. Inaccordance with the illuminant(s) identified for the scene, a thirdcapture parameter is derived wherein the third capture parameter isderived so as to obtain white balance in accordance with each identifiedilluminant. The third capture parameter is applied to the imagingassembly, and a final image of the scene is captured, or an additionaliteration is applied so as to fine-tune color balance of the scene.

Briefly, to identify regions in a scene that have different illuminants,a multi-stage technique may be applied, including a stage for imagepreview capture, a stage for analysis, a stage for image compensation,and a stage for final capture.

In the image preview capturing stage, the imaging system captures twoimages with two different sets of spectral sensitivities. In each setthe same spectral sensitivities may be used for all pixel of the imagingsensor frame. For example, on a half-press of the shutter the imagingsystem first captures a frame with RGB sensitivities for each pixel ofthe imaging sensor and sends it to an image buffer. Soon after theimaging system captures another image with sensitivities tuned now toR′G′B′, and this image is also sent to an image buffer.

In the analysis stage, there may be ratio calculations of the RGB andR′G′B′ outputs, such as calculation of r/g, r/b, r′/g′ and r′/b′, andsuch ratios may be calculated for each pixel. Based for example onexpectations for ranges of ratios, which may be predetermined bycalibration and pre-stored in a look-up table (LUT), an illuminant isdetermined for each pixel of the scene or for each region of pluralpixels of the scene. Image segmentation is thereafter applied, such thatclusters of pixels with similar ratios are grouped together based on therationale that pixels with similar ratios correspond to same illuminant,and the image is segmented into regions.

In the image compensation stage, white balance is compensated for thescene, including white balance compensation on a region-by-region basis.For each set of clustered pixels (corresponding to one illuminant) a newset of sensitivities to compensate the color shift due to the illuminantis determined, e.g. using the reciprocal of the ratios as weights indetermining new sensitivities or using the spectral power distributionof estimated illuminant as a weight to calculate new sensitivities. Arevised spatial mask is derived, wherein the magnitude of peaks andwavelength shifts of the sensitivity is related to particular values forthe spatial mask by a look-up-table. The same parameters are ordinarilyset for each cluster of pixels, perhaps varied by a pre-calibrated maskwhich compensates for pixel-to-pixel non-uniformities.

In the final capture stage, the revised spatial mask is applied to theimaging assembly, and an image of the scene is captured and recorded ina memory unit.

In more detail, a first capture of the scene is made with a firstcapture parameter. One such capture parameter is illustrated in FIG. 8,which shows capture parameters DR1, DG1 and DB1 which are applieduniformly in a spatial mask to all pixels of the image sensor 14 in theimaging assembly, so as to result in the spectral sensitivities shown inFIG. 8. These spectral sensitivities as shown in FIG. 8 may berepresented by matrix D_(λ) _(—) _(RGB) with dimensions 3 (correspondingto 3 channels) by m, where m is the number of wavelength samples. Forexample, if the visible light range is sampled from 400 nm to 700 nm inintervals of 5 nm, m=61. For simplicity consider that the contributionfrom the optics is also included in the spectral sensitivities D_(λ)_(—) _(RGB).

Consider the spectral power radiance of two illuminants: typicaltungsten-halogen illumination represented by matrix S_(λ) _(—) _(A)shown in FIG. 9 and typical daylight illumination represented by matrixS_(λ) _(—) _(D) shown in FIG. 10. Matrices S_(λ) _(—) _(A) and S_(λ)_(—) _(D)) have dimension 1 by m, where m is the number of wavelengthsamples.

Now, consider the spectral reflectances of two materials: a white sheetof paper (without fluorescent material) represented by matrix R_(λ) _(—)_(W) as shown in FIG. 11 and a yellow sheet of paper represented bymatrix R_(λ) _(—) _(Y) as shown in FIG. 12. Matrices R_(λ) _(—) _(W) andR_(λ) _(—) _(Y) have dimensions 1 by m, where m is the number ofwavelength samples. When a digital camera with spectral sensitivitiesD_(λ) _(—) _(RGB) captures a yellow paper with spectral reflectanceR_(λ) _(—) _(Y) illuminated by a daylight illuminant with spectralradiance S_(λ) _(—) _(D) the signals are integrated to produce threedigital signals represented by the following equation:C _(RGB) _(—) _(D) _(—) _(Y) =D _(λ) _(—) _(RGB)*diag(S _(λ) _(—)_(D))*R ^(T) _(λ) _(—) _(Y)  Equation 1

wherein C_(RGB) _(—) _(D) _(—) _(Y) has dimensions 3 by 1, and wherediag( ) denotes a diagonalized matrix and T denotes transposed matrix orvector.

Consider further when the same digital camera whose sensor is tuned tothe same spectral sensitivities D_(λ) _(—) _(RGB) captures a white paperwith spectral reflectance R_(λ) _(—) _(W) illuminated by atungsten-halogen illuminant with spectral radiance S_(λ) _(—) _(A). Asbefore, the signals are integrated to produce three digital signalsrepresented by the following equation:C _(RGB) _(—) _(A) _(—) _(W) =D _(λ) _(—) _(RGB)*diag(S _(λ) _(—)_(A))*R ^(T) _(λ) _(—) _(W)  Equation 2

wherein as before C_(RGB) _(—) _(A) _(—) _(W) has dimensions 3 by 1, anddiag( ) denotes a diagonalized matrix and T denotes transposed matrix orvector.

The two scenes captured as described above will have nearly identicalvalues for pixels representing the two sheets of paper—yellow paperunder daylight illumination in one case and white paper undertungsten-halogen in the other—as can be understood from the following,using examples of representative numerical values.

The red/green and red/blue ratios are calculated for both yellow paperunder daylight and white paper under tungsten-halogen illuminant.r/g _(—) D _(—) Y=C _(RGB) _(—) _(D) _(—) _(Y)(1)/C _(RGB) _(—) _(D)_(—) _(Y)(2)=1.6  Equation 3r/b _(—) D _(—) Y=C _(RGB) _(—) _(D) _(—) _(Y)(1)/C _(RGB) _(—) _(D)_(—) _(Y)(3)=2.9  Equation 4r/g _(—) A _(—) W=C _(RGB) _(—) _(A) _(—) _(W)(1)/C _(RGB) _(—) _(A)_(—) _(W)(2)=1.6  Equation 5r/b _(—) A _(—) W=C _(RGB) _(—) _(A) _(—) _(W)(1)/C _(RGB) _(—) _(A)_(—) _(W)(3)=2.9  Equation 6

In these equations, the parenthesized indices (i.e., (1), (2) and (3))respectively refer to the three outputs at each pixel of the imagesensor, namely, the output of the red-like channel of the image sensor,the output of the green-like channel of the image sensor, and the outputof the blue-like channel of the image sensor.

As a result, since r/g_D_Y=r/g_A_W and r/b_D_Y=r/b_A_W from thestatistics of the scene it is not possible to unambiguously identify thecorrect illuminant to perform color balance.

In this example embodiment, which uses a digital camera with tunablespectral sensitivities that allows reading of three digital signals perpixel, when the shutter is half-pressed two pre-programmed voltage masksare sequentially applied. With the first voltage mask, the RGB channelsof the sensor are set to spectral sensitivities shown in FIG. 8 andrepresented by D_(λ) _(—RGB) . The second voltage mask sets-up modifiedR′G′B′channels to the sensor with spectral sensitivities shown in FIG.13 and represented by D_(λ) _(—) _(R′G′B′).

In this embodiment, the spectral sensitivities caused by D_(λ) _(—)_(R′G′B′), are blue-shifted relative to those caused by D_(λ) _(—)_(RGB). More generally, the first and second capture parameters aresufficiently different from each other so as to provide gooddiscrimination for the imaging property of interest. For example, thefirst and second capture parameters may differ from each other such thatfirst and second image data captured under a variety of differentilluminants differs by more than a threshold value as between eachdifferent pair of illuminants. In some embodiments, as here, one set ofcapture parameters might cause a blue-shift in spectral sensitivityrelative to the other set of capture parameters, which tends to providegood discrimination between the broad spectrum of a scene illuminated bydaylight, the relatively red spectrum of a scene illuminated by atungsten-halogen illuminant, and the relatively narrow-band spectra ofilluminants such as fluorescent and sodium vapor illuminants, which tendto be narrow-band in the green area of the spectrum.

It should be recognized that the scene under consideration here is beingused for purposes of explanation. As a consequence, the scene is asomewhat extreme example, namely, a scene which contains both yellowpaper under white light and white paper under yellow light. For such ascene which contains such an extreme example, to obtain gooddiscrimination between illuminants and between regions having similarand different illuminants, the first and second capture parameter differgreatly, as can be understood by comparison for FIG. 8 to FIG. 13. Inmany practical situations, however, such an extreme example of a sceneis unlikely to be encountered often. Accordingly, in other embodiments,particularly those designed for application to somewhat moreconventionally-encountered scenes, the differences between the first andsecond capture parameters might not necessarily be as large as thoseshown here.

When the yellow paper with spectral reflectance R_(λ) _(—) _(Y) isimaged by the sensor with spectral sensitivities D_(λ) _(—) _(R′G′B′),under daylight illuminant with spectral S_(λ) _(—) _(D), three digitalsignals are obtained as represented by the following equationsC _(R′G′B′) _(—) _(D) _(—) _(Y) =D _(λ) _(—) _(R′G′B′)*diag(S _(λ) _(—)_(D))*R ^(T) _(λ) _(—) _(Y)  Equation 7

In the same way, when the white paper with spectral reflectance R_(λ)_(—) _(W) is imaged by the sensor with spectral sensitivities D_(λ) _(—)_(R′G′B′) under tungsten-halogen illuminant with S_(λ) _(—) _(A), threedigital signals are obtained as represented by the following equation:C _(R′G′B′) _(—) _(A) _(—) _(W) =D _(λ) _(—) _(R′G′B′)*diag(S _(λ) _(—)_(A))*R ^(T) _(λ) _(—) _(W)  Equation 8

The red/green and red/blue ratios are calculated for both yellow paperunder daylight and white paper under tungsten-halogen illuminant, andyield the following, again using examples of representative numericalvalues:r′/g′ _(—) D _(—) Y=C _(R′G′B′) _(—) _(D) _(—) _(Y)(1)/C _(R′G′B′) _(—)_(D) _(—) _(Y)(2)=0.9  Equation 9g′/b′ _(—) D _(—) Y=C _(R′G′B′) _(—) _(D) _(—) _(Y)(2)/C _(R′G′B′) _(—)_(D) _(—) _(Y)(3)=1.2  Equation 10r′/g′ _(—) A _(—) W=C _(R′G′B′) _(—) _(A) _(—) _(W)(1)/C _(R′G′B′) _(—)_(A) _(—) _(W)(2)=0.8  Equation 11g′/b′ _(—) A _(—) W=C _(R′G′B′) _(—) _(A) _(—) _(W)(2)/C _(R′G′B′) _(—)_(A) _(—) _(W)(3)=1.4  Equation 12

A look-up-table with channel ratios r/g=1.6, r/b=2.9, r′/g′=0.9 andg′/b′=1.2 would indicate that there is high likelihood that theilluminant is daylight while r/g=1.6, r/b=2.9, r′/g′=0.8 and r′b′=1.4would indicate that there is high likelihood that the illuminant istungsten-halogen.

More precisely, as previously discussed, because the numerical resultsof Equations 3 and 4 are identical to those of Equations 5 and 6, it isnot possible with a single capture using D_(λ) _(—) _(RGB) tounambiguously differentiate between a daylight illuminant on yellowpaper and a tungsten-halogen illuminant on white paper. This singlecapture is thus unable to unambiguously identify the correct illuminantfor this example of a scene. By comparison of the first capture with asecond captured with a different capture parameter, however, it becomespossible to identify the illuminant. A suitable LUT might containentries as follows:

TABLE 1 LUT Entries Capture 1 Capture 2 r/g r/b r/g r/b Illuminant 1.62.9 0.9 1.2 Daylight 1.6 2.9 0.8 1.4 Tungsten- Halogen

This example illustrates a case in which the use of the imaging sensorwith two captures of images with two different sets of tuned spectralsensitivities. In this case, the two captures with different captureparameters increased the degrees of freedom, providing more metrics thatcan unambiguously determine the type of illuminant, thereby increasingthe robustness of the color balancing method. Once the illuminant isidentified, the spectral sensitivities of the imaging assembly are tunedby determining an updated capture parameter, wherein the updated captureparameter compensates for the color cast provided by the illumination.

For example, for a color balanced scene, r/g and r/b should be close toone for a gray or white (neutral) color. The new spectral sensitivitiesto be tuned D_(λ) _(—) _(New) are selected in order to produce thisresponse. Considering that the illuminant is tungsten-halogen, in matrixnotation there are now two equations as follows:D _(λ) _(—) _(New)(1,:)*diag(S _(λ) _(—) _(A))*R ^(T) _(λ) _(—) _(W) =D_(λ) _(—) _(New)(2,:)*diag(S _(λ) _(—) _(A))*R ^(T) _(λ) _(—)_(W)  Equation 13D _(λ) _(—) _(New)(1,:)*diag(S _(λ) _(—) _(A))*R ^(T) _(λ) _(—) _(W) =D_(λ) _(—) _(New)(3,:)*diag(S _(λ) _(—) _(A))*R ^(T) _(λ) _(—)_(W)  Equation 14

Since there are multiple unknowns (3×m) and only two equations thesystem is an undetermined system that has multiple possible solutions.However, it is necessary to impose smoothness, continuity constraintsand consider a set of spectral sensitivities curves that is within therange of sensor spectral tunability. There are several numerical methodsto find a suitable spectral sensitivity with these constraints. There isa limited possibility for types of illumination and a look-up-table caneasily be generated giving the color sensitivities necessary for colorbalancing for each illuminant.

On the other hand, it is also possible to come up with a simpleestimation procedure by using the reciprocal of r/g and r/b ratios asweights to generate new spectral sensitivities based on D_(λ) _(—)_(RGB) sensitivities.

For the example above, after identifying the illuminant, a new captureparameter is derived for good color balance and to avoid color cast. Onepossible spectral sensitivity D_(λ) _(—) _(New) that eliminates thecolor cast is given by the spectral sensitivities shown in FIG. 14. Anelectronic mask is generated with voltages that produce the spectralsensitivities shown in FIG. 14, as pre-determined by calibration and theimage is captured when shutter is pressed and saved in a memory unit.Using reasoning similar to that explained above, digital signals areobtained by using the spectral sensitivity D_(λ) _(—) _(New), asfollows:C _(RGB) _(—) _(New) _(—) _(A) _(—) _(W) =D _(λ) _(—) _(New)*diag(S _(λ)_(—) _(A))*R ^(T) _(λ) _(—) _(W)  Equation 15

When the r/g and r/b ratios are calculated with the digital signalsobtained, it is observed that r/g=r/b=1 for the white neutral region,thus proving the effectiveness of the color balance.

In another embodiment the detection of region of multiple illuminants byanalyzing r/g, r/b, r′/g′ and r′/b′ ratios is considered. Based on theilluminant detected for each region, color sensitivity compensation isused for each illuminated region with difference correlated temperatureand the color compensations are transcribed as the voltages for eachregion in the spatial electrical mask that will be applied to thetunable sensitivity imaging sensor during capture of the scene. In thisembodiment, each region that has illuminant with distinct correlatedcolor temperature will be compensated in one single shot.

Some scenes have multiple regions each illuminated by a differentilluminant. In such a situation, embodiments herein expand the conceptof global ratio values to regions of ratio values in which when a widerange of objects with distinct spectral reflectances are imaged by thesensor under a specific illumination the ratios can be clustered in oneregion. This region of clustered ratios will be distinct to same objectsilluminated by a different illuminant using the same sensor. Bydetermining ratios (by calculating the ratios of every pixel or bysampling spatially ratios of an image) it is possible to estimate theilluminant by looking into which cluster of ratios the sample belongs.

Consider an example described in the Summary herein, wherein a scene hasdifferent illuminants in the left and right halves, such as a scene inwhose left half a yellow paper is illuminated by daylight illuminantwith blue color cast while in its right half a white paper isilluminated by halogen lamp with a yellow cast. For this example, afirst image is captured using first capture parameters shown in FIG. 8(i.e., D_(λ) _(—) _(RGB)), and a second image is captured using secondcapture parameters shown in FIG. 13 (i.e., D_(λ) _(—) _(R′G′B′)).

For the first capture parameters of D_(λ) _(—) _(RGB), the ratios of agray background (signified by G) surrounding the yellow paper (Y) underdaylight illumination (D) in the left hand side, and the ratios of adarker background (signified by K) surrounding the white paper (W)illuminated by tungsten/halogen lamp (A), are computed from measurementsin the same fashion as performed for the yellow and white papers. Usingexamples of representative numerical values, the resulting ratios forthe gray surround under daylight illumination are shown in the equationsbelow. In these equations, GY refers to the gray background surroundingthe yellow paper under daylight illumination D, and GK refers to thedarker background surrounding the white paper illuminated bytungsten/halogen lamp A:r/g _(—) D _(—) GY=C _(RGB) _(—) _(D) _(—) _(GY)(1)/C _(RGB) _(—) _(D)_(—) _(GY)(2)=0.8  Equation 16r/b _(—) D _(—) GY=C _(RGB) _(—) _(D) _(—) _(GY)(1)/C _(RGB) _(—) _(D)_(—) _(GY)(3)=0.8  Equation 17

On the other hand, again using examples of representative numericalvalues, the resulting ratios for the gray surround undertungsten/halogen illumination are:r/g _(—) A _(—) GK=C _(RGB) _(—) _(A) _(—) _(GK)(1)/C _(RGB) _(—) _(A)_(—) _(GK)(2)=2.6  Equation 18r/b _(—) A _(—) GK=C _(RGB) _(—) _(A) _(—) _(GK)(1)/C _(RGB) _(—) _(A)_(—) _(GK)(3)=5.4  Equation 19

For the second capture parameters of D_(λ) _(—) _(R′G′B′), for the graysurround under daylight illumination, the ratios of r′/g′ and g′/b′ are:r′/g′ _(—) D _(—) GY=C _(R′G′B′) _(—) _(D) _(—) _(GY)(1)/C _(R′G′B′)_(—) _(D) _(—) _(GY)(2)=0.8  Equation 20g′/b′ _(—) D _(—) GY=C _(R′G′B′) _(—) _(D) _(—) _(GY)(2)/C _(R′G′B′)_(—) _(D) _(—) _(GY)(3)=0.9  Equation 21

For the darker surround under tungsten-halogen illumination, the ratiosare:r′/g′ _(—) A _(—) GK=C _(R′G′B′) _(—) _(A) _(—) _(GK)(1)/C _(R′G′B′)_(—) _(A) _(—) _(GK)(2)=0.6  Equation 22g′/b′ _(—) A _(—) GK=C _(R′G′B′) _(—) _(A) _(—) _(GK)(2)/C _(R′G′B′)_(—) _(A) _(—) _(GK)(3)=1.3  Equation 23

FIG. 15 shows a plot of r/g by r/b ratios for the four objects (whitepaper under tungsten/halogen from Equations 5 and 6; gray backgroundunder daylight from Equations 16 and 17; yellow paper under daylightfrom Equations 3 and 4; and darker background under tungsten/halogenfrom Equations 18 and 19) using the first capture parameters D_(λ) _(—)_(RGB) whose spectral sensitivity is represented in FIG. 8. Note thatthe white r/b and r/g values coincide for both white paper undertungsten/halogen and yellow paper under daylight. The bluish nature ofdaylight illuminant and the relative low red content of this illuminantproduce relatively low r/b and r/g ratios compared to a tungsten/halogenillumination that has lots of energy in the red region of the spectraand not much in the blue region of the spectra as illustrated in FIG.15. There is an area of overlap in which it is undistinguishable whichis the illuminant because of the overlap between the ellipsesrepresenting the clusters of [r/g r/b] pairs for each illumination.

FIG. 16 shows a plot of r′/g′ by g′/b′ ratios for the four objects(white paper under tungsten/halogen from Equations 11 and 12; graybackground under daylight from Equations 20 and 21; yellow paper underdaylight from Equations 9 and 10; and darker background undertungsten/halogen from Equations 22 and 23) using second captureparameters D_(λ) _(—) _(R′G′B′) whose spectral sensitivity is shown inFIG. 13. In this graphical representation there is no overlap betweenratios for both illuminants resulting in a more unequivocal estimationof illumination.

Moreover, because of the clustering which is evident from thesecomparisons of first and second captures, it is also apparent that theregions of the image can be distinguished, as between first and secondregions illuminated by respectively different illuminants. Thus, in thecase of the scene under consideration in this example, it becomespossible to identify a first region on the left half which isilluminated by daylight illuminant, and a second region on the righthalf illuminated by a tungsten-halogen illuminant.

In the embodiments described herein, the tunable imaging assembly maytunable such that each pixel or each region of multiple pixels istunable individually, such that the spectral responsivity of each pixelor region of pixels is tunable independently of the spectralresponsivity of other pixels or regions of pixels. In some exampleembodiments, the entirety of the imaging assembly may be tuned to thesame spectral responsivity, such that substantially all pixels andsubstantially all regions of pixels are tuned to substantially the samespectral responsivity. Such an arrangement might be advantageous insituations where a single imaging property is applicable to nearly theentirety of the scene, such as a single white balance or a singledynamic range.

To assist in a greater understanding of the disclosure, a brief summaryand some further examples are provided here for the analysis stage.

In the analysis stage, objects in a scene are decoupled from theirillumination. This is generally difficult because of the inherentambiguity of captured information. For example, there is need to know ifthe captured object in the scene is actually yellow or if it is a whiteobject under yellowish light. Human visual system automaticallycompensates by equalizing color bias in lighting by performing anadaptation based on physiological responses in the visual systemcombined with cognitive information (for example, a human being knowsthat a red object with an apple shape is probably red, and is not merelya white object under red light). However, imaging sensors do not performthis compensation automatically. There is a need to perform some imageanalysis in order to identify the illumination (illumination estimation)and once the illumination estimation takes place it is possible tocompensate the color channels of the image in order to produce a morecolor balanced scene.

Conventional cameras are usually limited in their ability for extensiveimage analysis by a number of factors, including a global compensation,a limited number of fixed color channels, and a lack of flexibility. Byway of explanation, because of global compensation, wherein all pixelsare compensated the same, a global correction by these algorithms canonly address one illumination in the scene. Because of a limited numberof fixed color channels, wherein most color imaging sensors are based onthe traditional red-green-blue Bayer pattern, it is an ill-posed inverseproblem to estimate the illumination using only three fixed spatialcolor sensors. Because of lack of flexibility, imaging underillumination with a strong color saturation will typically saturate thesignals in one or more color channels. For example, imaging under anincandescent lamp may saturate the red color signal. As a result, aninaccurate analysis will ensue with an incorrect estimation ofilluminant. Even if the illuminant is accurately estimated the saturatedsignals will lead to incorrect color balancing.

According to embodiments described herein, the analysis stage can beapproached using a strategy of sequential adaptive filtering. In thisapproach the array of tunable filter sensors receives a sequence ofpre-programmed capture parameters that adjusts the color filters of allpixels with same values. For each voltage mask an image is captured. Ifn images are captured, n values are provided for each pixel, whereineach of the n values has a red-like channel, a green-like channel and ablue-like channel. The imaging system may be pre-calibrated by capturingknown color targets under a variety of pre-determined illumination and amulti-dimensional look-up-table may be generated to map captured digitalvalues (for each filter adjustment) to particular illuminants. Themulti-dimensional look-up-table is then used to estimate theillumination for each pixel of the image.

In one typical arrangement, the illumination will be described in termsof correlated-color-temperature (CCT). For example, a very reddishhorizontal sunlight illumination typical of sunsets will have a CCT of2,000 K. An incandescent light bulb will have a CCT of 2,900 K. Naturaldaylight will depend on atmospheric conditions but it will be in therange of CCTs 4,500K to 6,500 K. Artificial daylight simulation lampsare also in the same range of CCTs as natural daylight. Very dark blueskies after sunset produce a very high CCT (10,000 to 20,000 K).Typically, white balance adjusts colors to correspond to a look underdaylight illumination. Therefore if the illumination is 3,000 K thesystem might apply a bias of +2,500 K to compensate to a preset value of5,500 K. If the illumination is 10,000 K, the system might apply a biasof −4,500 K to a preset value of 5,500 K.

If one channel or more channels are saturated the analysis process isrepeated with new color filter adjustments.

In the capture stage of image enhancement mode the main purpose is toautomatically enhance the image based on analysis of the scene (in thisexample white balance) and perform pixel-based adjustments.

The calculated bias spatial mask given by the analysis stage may beconverted to a voltage mask of voltages for application to controlelectrodes of the image sensor using a pre-calculated look-up-table(based on calibration) to apply the correct voltages for the tunablefilter of each pixel, and an image superior in white balance iscaptured.

A high-level summary of embodiments described herein thus proceeds asfollows:

Step 1. Capture sequence of images with pre-set color filters in thetunable filter imaging sensor

Step 2. Analyze if there are no saturated channels and if yes, selectnew filters and go to step 1.

Step 3. Compute illumination estimation for each pixel in the image byusing the multi-dimensional look-up table

Step 4. Compute CCT bias based on estimated illumination and pre-set CCTfor white balance correction producing spatial CCT-bias mask

Step 5. Compute spatial voltage mask based on a pre-calculatedconversion look-up-table from CCT bias values to voltages

Step 6. Apply spatial voltage mask to tune filters and capture images.

<FIGS. 17 and 18>

FIG. 17 is a block diagram showing another example embodiment of anarrangement of a digital camera 200. In the embodiment of FIG. 17, partsand features that are largely similar to those of the example embodimentof FIG. 1 are illustrated with like reference numerals, and a detailedexplanation thereof is omitted in the interest of brevity.

One way that the embodiment of FIG. 17 differs from the embodiment ofFIG. 1 concerns the construction of the tunable imaging assembly. In theembodiment of FIG. 1, the tunable imaging assembly includes tunableimage sensor 14, perhaps in combination with optics such as lens 10.Because the image sensor 14 in the embodiment of FIG. 1 itself has atunable spectral response, it is customary to omit a preceding colorfilter array, since the inclusion of any filter necessarily woulddecrease the signal-to-noise ratio by filtering the amount of lightincident on image sensor 14.

In contrast, in the embodiment of FIG. 17, the spectral responsivity ofimage sensor 214 is not necessarily tunable, but rather the spectralresponsivity of a preceding color filter array 219 is. Thus, in theexample embodiment of FIG. 17, the tunable imaging assembly includestunable color filter array (CFA) 219 and image sensor 214, perhaps incombination with optics such as lens 10. In the embodiment of FIG. 17,image sensor 214 is not necessarily tunable, although in otherembodiments it might be.

Turning more specifically to the embodiment of FIG. 17, a light beam(light beam incident upon the angle of view of the lens) from an objectin a scene that goes through the optical system (image sensing lens) 10passes through an opening of a shutter 12 having a diaphragm function,is filtered by tunable color filter array 219, and forms an opticalimage of the object on the image sensing surface of image sensor 214.The image sensor 214 converts the optical image to analog image signalsand outputs the signals to an A/D converter 16. The A/D converter 16converts the analog image signal to digital image signals (image data).

In FIG. 17, an imaging assembly is comprised of tunable color filterarray 219 and image sensor 214 together with associated optics, suchthat in some embodiments the imaging assembly is comprised of imagesensor 214 preceded by color filter array 219 and lens 10.

Tunable color filter array 219 may be a spatial color filter array, suchas a color filter array having a spatial distribution of a repeatingpattern of filter elements. In this case, image data output from imagesensor 214 is demosaiced, so as to result in output of a red-likechannel for each pixel, a green-like channel for each pixel, and ablue-light channel for each pixel. Alternatively, tunable color filterarray 219 might be a temporal color filter array, in which case thecolor filter quickly and sequentially changes spectral responsivity,with image data collected by image sensor 214 after each change. In thiscase, the sequential outputs of image sensor 214 are collected so as toresult in output signals for each pixel for a red-like channel, agreen-like channel, and a blue-light channel.

The spectral responsivity of tunable color filter array 219 is tunablein accordance with a capture parameter 217. In this embodiment, captureparameter 217 may be comprised of multiple spatial masks, with one maskfor each channel of information output by image sensor 214, namely, theaforementioned red-like channel, green-like channel, and blue-lightchannel. Thus, in this example where image sensor 214 outputs three ormore channels, capture parameters 217 include a spatial mask DR for thered-like channel of information, a spatial mask DG for the green-likechannel of information, and a spatial mask DB for the blue-light channelof information. Each spatial mask comprises an array of controlparameters applied to the tunable color filter array 219 incorrespondence to pixels or regions of pixels in image sensor 214. Theresulting spectral responsivity of each pixel, or each region of pluralpixels, is thus tunable individually and independently of other pixelsor regions of pixels, by virtue of the capture parameter 217 imposed ontunable color filter array 219.

Tunable color filter array 219 may be comprised of a tunable colorfilter array as described in U.S. Pat. No. 6,466,961 by Miller,mentioned hereinabove. Spatial masks DR, DG and DB may correspond tovoltage biases applied to control electrodes of the tunable color filterarray 219.

FIG. 18 is a flow diagram for explaining operation of this exampleembodiment. The process steps shown in FIG. 18 are computer-executableprocess steps executed primarily by system controller 50 based oncomputer-executable process steps stored in a computer-readable memorymedium such as non-volatile memory 56.

Briefly, according to FIG. 18, an imaging property for a scene isidentified, such as a color balance property or a high-dynamic rangeproperty. Two images of the scene are captured with two different setsof capture parameters which tune the spectral responsivity of an imagingassembly such as an imaging assembly which includes a tunable colorfilter array which precedes an image sensor. The capture parameters maybe applied, and the two images of the scene captured, when a shutter ishalf-pressed. The two sets of capture parameters may be pre-designated,and the differences between them are designed to provide gooddiscrimination for the imaging property. The imaging property for thescene is identified based on a comparison of the two images. The imagingproperty may be identified individually for different regions of thescene, in which case the two captured images may be divided into pluralregions based on statistics of the images. Based on the imaging property(or properties) thus identified, a third capture parameter may bederived, wherein the third capture parameter is designed for capture ofthe scene with accuracy for the identified imaging property. The thirdcapture parameter is applied to the tunable imaging assembly, such as byapplication to a tunable color filter array that precedes an imagesensor, and a final image of the scene is captured, such as when ashutter is fully pressed.

In more detail, in step S1801, a first capture parameter is applied totunable color filter array 219. The first capture parameter may be apre-designated capture parameter stored in non-volatile memory 56. Inthis example embodiment, the capture parameter may be a spatial maskwhich individually tunes each pixel or each region of plural pixels intunable color filter array 219, such as by application of spatial masksDR, DG and DB.

Following application of the first capture parameter to tunable colorfilter array 219, a first image is captured in step S1802. Thereafter, adifferent capture parameter is applied to tunable color filter array219, and a second image is captured (steps S1803 and S1804). The secondcapture parameter may likewise be a pre-designated capture parameterstored in non-volatile memory 56. The second capture parameter isdifferent from the first capture parameter, and the differences aredesigned to provide good discrimination for the imaging property beingidentified.

As one example, the imaging property being identified might be theilluminant or illuminants for different regions in the scene. In such asituation, the first and second capture parameters are different so asto provide good discrimination between differing illuminants. Forexample, the first and second capture parameters may differ from eachother such that first and second image data captured under a variety ofdifferent illuminants differs by more than a threshold value as betweeneach different pair of illuminants. In some embodiments, as here, oneset of capture parameters might cause a blue-shift in spectralsensitivity relative to the other set of capture parameters, which tendsto provide good discrimination between the broad spectrum of a sceneilluminated by daylight, the relatively red spectrum of a sceneilluminated by a tungsten-halogen illuminant, and the relativelynarrow-band spectra of illuminants such as fluorescent and sodium vaporilluminants, which tend to be narrow-band in the green area of thespectrum.

After capture of the first and second images, step S1805 identifies theimaging property by comparison thereof. Such a comparison, and theresulting identification of the imaging property, might proceed in wayslargely similar to those described above.

Step S1806 derive a third capture parameter which accommodates theimaging property. For example, and as described above in connection withother embodiments, the imaging property of interest might be theilluminant or illuminants for the scene. In such a situation, the thirdcapture parameter is derived so as to accommodate the precise nature ofthe illuminant or illuminants identified in step S1805. Likewise, and asalso described above in connection with other embodiments, the imagingproperty of image might be the dynamic range for the scene. In such acase, the third capture parameter is derived so as to accommodate thedynamic range identified for the scene.

Steps S1807 and S1808 apply the third capture parameter to the tunablecolor filter array 219, and capture an additional image. If the image isacceptable in step S1809, then a final image is stored in step S1810. Onthe other hand, if the image is deemed not acceptable in step S1809,then a further iteration of the procedure is performed, by returning tostep S1805.

Acceptability of the image in step S1809 may be performed automaticallyby system controller 50 based on pre-designated tolerances. In otherembodiments, acceptability of the image in step S1809 might bedetermined by the user.

In the embodiments described herein, the tunable color filter array maytunable such that each pixel or each region of multiple pixels istunable individually, such that the spectral responsivity of each pixelor region of pixels is tunable independently of the spectralresponsivity of other pixels or regions of pixels. In some exampleembodiments, the entirety of the color filter array may be tuned to thesame spectral responsivity, such that substantially all pixels andsubstantially all regions of pixels are tuned to substantially the samespectral responsivity. Such an arrangement might be advantageous insituations where a single imaging property is applicable to nearly theentirety of the scene, such as a single white balance or a singledynamic range.

Other Embodiments

According to other embodiments contemplated by the present disclosure,example embodiments may include a computer processor such as a singlecore or multi-core central processing unit (CPU) or micro-processingunit (MPU), which is constructed to realize the functionality describedabove. The computer processor might be incorporated in a stand-aloneapparatus or in a multi-component apparatus, or might comprise multiplecomputer processors which are constructed to work together to realizesuch functionality. The computer processor or processors execute acomputer-executable program (sometimes referred to ascomputer-executable instructions or computer-executable code) to performsome or all of the above-described functions. The computer-executableprogram may be pre-stored in the computer processor(s), or the computerprocessor(s) may be functionally connected for access to anon-transitory computer-readable storage medium on which thecomputer-executable program or program steps are stored. For thesepurposes, access to the non-transitory computer-readable storage mediummay be a local access such as by access via a local memory busstructure, or may be a remote access such as by access via a wired orwireless network or Internet. The computer processor(s) may thereafterbe operated to execute the computer-executable program or program stepsto perform functions of the above-described embodiments.

According to still further embodiments contemplated by the presentdisclosure, example embodiments may include methods in which thefunctionality described above is performed by a computer processor suchas a single core or multi-core central processing unit (CPU) ormicro-processing unit (MPU). As explained above, the computer processormight be incorporated in a stand-alone apparatus or in a multi-componentapparatus, or might comprise multiple computer processors which worktogether to perform such functionality. The computer processor orprocessors execute a computer-executable program (sometimes referred toas computer-executable instructions or computer-executable code) toperform some or all of the above-described functions. Thecomputer-executable program may be pre-stored in the computerprocessor(s), or the computer processor(s) may be functionally connectedfor access to a non-transitory computer-readable storage medium on whichthe computer-executable program or program steps are stored. Access tothe non-transitory computer-readable storage medium may form part of themethod of the embodiment. For these purposes, access to thenon-transitory computer-readable storage medium may be a local accesssuch as by access via a local memory bus structure, or may be a remoteaccess such as by access via a wired or wireless network or Internet.The computer processor(s) is/are thereafter operated to execute thecomputer-executable program or program steps to perform functions of theabove-described embodiments.

The non-transitory computer-readable storage medium on which acomputer-executable program or program steps are stored may be any of awide variety of tangible storage devices which are constructed toretrievably store data, including, for example, any of a flexible disk(floppy disk), a hard disk, an optical disk, a magneto-optical disk, acompact disc (CD), a digital versatile disc (DVD), micro-drive, a readonly memory (ROM), random access memory (RAM), erasable programmableread only memory (EPROM), electrically erasable programmable read onlymemory (EEPROM), dynamic random access memory (DRAM), video RAM (VRAM),a magnetic tape or card, optical card, nanosystem, molecular memoryintegrated circuit, redundant array of independent disks (RAID), anonvolatile memory card, a flash memory device, a storage of distributedcomputing systems and the like. The storage medium may be a functionexpansion unit removably inserted in and/or remotely accessed by theapparatus or system for use with the computer processor(s).

This disclosure has provided a detailed description with respect toparticular representative embodiments. It is understood that the scopeof the appended claims is not limited to the above-described embodimentsand that various changes and modifications may be made without departingfrom the scope of the claims.

What is claimed is:
 1. A method for identifying an imaging property fora scene by using an imaging assembly which has a spectral response whichis tunable in accordance with a capture parameter, the methodcomprising: applying a first capture parameter to the imaging assembly,and capturing a first image of the scene using the imaging assemblywhose spectral response is tuned in accordance with the first captureparameter; applying a second capture parameter to the imaging assembly,wherein the second capture parameter is different from the first captureparameter, and capturing a second image of the scene using the imagingassembly whose spectral response is tuned in accordance with the secondcapture parameter; and identifying an illuminant for the scene based onat least one color channel ratio of the first image of the scene and atleast one color channel ratio of the second image of the scene, whereinthe illuminant has a defined spectral power distribution.
 2. The methodaccording to claim 1, wherein in said identifying step, the first andsecond images are compared to identify multiple regions of the scenehaving a similar illuminant.
 3. The method according to claim 1, whereinin said identifying step, the first and second images are compared toidentify multiple regions in the scene having illuminants that differfrom one another.
 4. The method according to claim 1, wherein the firstand second capture parameters are sufficiently different from each otherso as to provide good discrimination for the illuminant.
 5. The methodaccording to claim 1, wherein one of the first and second captureparameters causes a blue-shift in spectral sensitivity relative to theother of the first and second capture parameters.
 6. The methodaccording to claim 1, wherein the imaging assembly provides three ormore channels of information for each pixel of the scene, including ared-like channel, a blue-like channel and a green-like channel, andwherein in said comparing step, the first and second images of the sceneare compared by using ratios of the three color channels.
 7. The methodaccording to claim 1, wherein the imaging assembly comprises an imagesensor which has a tunable spectral response.
 8. The method according toclaim 1, wherein the imaging assembly comprises an image sensor and apreceding color filter array which has a tunable spectral response.
 9. Amethod for adjusting white balance of an image capture device whichincludes an imaging assembly having a spectral response which is tunablein accordance with a capture parameter, the method comprising: applyinga first capture parameter to the imaging assembly; capturing a firstimage of the scene using the imaging assembly whose spectral response istuned in accordance with the first capture parameter; applying a secondcapture parameter to the imaging assembly, wherein the second captureparameter is different from the first capture parameter; capturing asecond image of the scene by using the imaging assembly whose spectralresponse is tuned in accordance with the second capture parameter;determining an illuminant for the scene based on at least one colorchannel ratio of the first image and at least one color channel ratio ofthe second image of the scene, wherein the illuminant has a definedspectral power distribution; deriving a third capture parameter based onthe defined spectral power distribution of the illuminant, wherein thethird capture parameter is derived so as to obtain white balance inaccordance with the defined spectral power distribution of theilluminant.
 10. The method according to claim 9, wherein in saiddetermining step, the first and second images are compared to identifymultiple regions of the scene having a similar illuminant, and whereinthe third capture parameter is applied to the imaging assembly for allof such multiple regions.
 11. The method according to claim 9, whereinin said determining step, the first and second images are compared toidentify multiple regions in the scene having illuminants that differfrom one another, and further comprising the step of deriving multiplethird capture parameters, one each for each of such multiple regions,and wherein each such third capture parameter is applied to the imagingassembly in correspondence to each of such multiple regions.
 12. Themethod according to claim 9, wherein one of the first and second captureparameters causes a blue-shift in spectral sensitivity relative to theother of the first and second capture parameters.
 13. The methodaccording to claim 9, wherein the imaging assembly provides two or morecolor channels of information for each pixel of the scene, and whereinin said determining step, the illuminant is determined based on ratiosof the two or more color channels.
 14. The method according to claim 13,wherein determining an illuminant for the scene includes referencing alook-up table based on the ratios of the channels, wherein the look-uptable includes entries that associate two or more ratios of the colorchannels with illuminants.
 15. The method according to claim 14, whereinthe color channels include a red-like channel, a blue-like channel, anda green-like channel, and wherein the ratios include a red-like channelto green-like channel ratio and a red-like channel to blue-like channelratio.
 16. The method according to claim 9, wherein the illuminant istungsten, halogen, fluorescent, sodium vapor, or a daylight illuminant.17. An apparatus comprising: an imaging assembly having a spectralresponse which is tunable in accordance with a capture parameter; acontrol unit constructed to apply a first capture parameter and a secondcapture parameter to the imaging assembly, and to capture respectiveones of a first image of the scene and a second image of the scene usingthe imaging assembly whose spectral response is tuned in accordance withthe first and second capture parameters, wherein the second captureparameter is different from the first capture parameter; and a sceneproperty analysis module constructed to determine an illuminant for thescene based on at least one color channel ratio of the first image ofthe scene and at least one color channel ratio of the second image ofthe scene, wherein the illuminant has a defined spectral powerdistribution.
 18. The apparatus according to claim 17, wherein the firstand second images are compared to identify multiple regions of the scenehaving a similar illuminant.
 19. The apparatus according to claim 17,wherein the first and second images are compared to identify multipleregions in the scene having illuminants that differ from one another.20. The apparatus according to claim 17, wherein the first and secondcapture parameters are sufficiently different from each other so as toprovide good discrimination for the illuminant.
 21. The apparatusaccording to claim 17, wherein one of the first and second captureparameters causes a blue-shift in spectral sensitivity relative to theother of the first and second capture parameters.
 22. The apparatusaccording to claim 17, wherein the imaging assembly provides three ormore channels of information for each pixel of the scene, including ared-like channel, a blue-like channel and a green-like channel.
 23. Theapparatus according to claim 17, wherein the imaging assembly comprisesan image sensor which has a tunable spectral response.
 24. The apparatusaccording to claim 17, wherein the imaging assembly comprises an imagesensor and a preceding color filter array which has a tunable spectralresponse.
 25. An apparatus comprising: an imaging assembly having aspectral response which is tunable in accordance with a captureparameter; a control unit constructed to apply a first capture parameterand a second capture parameter to the imaging assembly, and to capturerespective ones of a first image of the scene and a second image of thescene using the imaging assembly whose spectral response is tuned inaccordance with the first and second capture parameters, wherein thesecond capture parameter is different from the first capture parameter;a scene property analysis module constructed to determine an illuminantfor the scene based on at least one color channel ratio of the firstimage of the scene and at least one color channel ratio of the secondimage of the scene, wherein the illuminant has a defined spectral powerdistribution; and a mask generation unit constructed to derive a thirdcapture parameter in accordance with the illuminant identified for thescene, wherein the third capture parameter is derived so as to obtainwhite balance in accordance with the identified illuminant; wherein thecontrol unit is further constructed to apply the third capture parameterto the imaging assembly.
 26. The apparatus according to claim 25,wherein the at least one color channel ratio of the first image and theat least one color channel ratio of the second image are compared toidentify multiple regions of the scene having a similar illuminant, andwherein the third capture parameter is applied to the imaging assemblyfor all of such multiple regions.
 27. The apparatus according to claim25, wherein the at least one color channel ratio of the first image andthe at least one color channel ratio of the second image are compared toidentify multiple regions in the scene having illuminants that differfrom one another, wherein the mask generation unit derives multiplethird capture parameters, one each for each of such multiple regions,and wherein each such third capture parameter is applied to the imagingassembly in correspondence to each of such multiple regions.
 28. Theapparatus according to claim 25, wherein the first and second captureparameters are sufficiently different from each other so as to providegood discrimination for the illuminant.
 29. The apparatus according toclaim 25, wherein one of the first and second capture parameters causesa blue-shift in spectral sensitivity relative to the other of the firstand second capture parameters.
 30. The apparatus according to claim 25,wherein the imaging assembly provides three or more channels ofinformation for each pixel of the scene, including a red-like channel, ablue-like channel and a green-like channel.
 31. The apparatus accordingto claim 25, wherein the imaging assembly comprises an image sensorwhich has a tunable spectral response.
 32. The apparatus according toclaim 25, wherein the imaging assembly comprises an image sensor and apreceding color filter array which has a tunable spectral response. 33.A non-transitory computer-readable medium having computer-executableprocess steps stored thereon for identifying an imaging property for ascene by using an imaging assembly which has a spectral response whichis tunable in accordance with a capture parameter, wherein said processsteps comprise: a first applying step to apply a first capture parameterto the imaging assembly, and capturing a first image of the scene usingthe imaging assembly whose spectral response is tuned in accordance withthe first capture parameter; a second applying step to apply a secondcapture parameter to the imaging assembly, wherein the second captureparameter is different from the first capture parameter, and capturing asecond image of the scene using the imaging assembly whose spectralresponse is tuned in accordance with the second capture parameter; andan identifying step to identify an illuminant for the scene based on atleast one color channel ratio of the first image of the scene and atleast one color channel ratio of the second image of the scene, whereinthe illuminant has a defined spectral power distribution.
 34. Thenon-transitory computer-readable medium according to claim 33, whereinsaid process steps further comprise a comparing step, wherein the firstand second images are compared to identify multiple regions of the scenehaving a similar illuminant.
 35. The non-transitory computer-readablemedium according to claim 33, wherein said process steps furthercomprise a comparing step, wherein the first and second images arecompared to identify multiple regions in the scene having illuminantsthat differ from one another.
 36. The non-transitory computer-readablemedium according to claim 33, wherein the first and second captureparameters are sufficiently different from each other so as to providegood discrimination for the illuminant.
 37. The non-transitorycomputer-readable medium according to claim 33, wherein one of the firstand second capture parameters causes a blue-shift in spectralsensitivity relative to the other of the first and second captureparameters.
 38. The non-transitory computer-readable medium according toclaim 33, wherein the imaging assembly provides three or more channelsof information for each pixel of the scene, including a red-likechannel, a blue-like channel and a identified based at least on ared/green ratio and a red/blue ratio of the first image and a red/greenratio and a red/blue ratio of the second image.
 39. The non-transitorycomputer-readable medium according to claim 33, wherein the imagingassembly comprises an image sensor which has a tunable spectralresponse.
 40. The non-transitory computer-readable medium according toclaim 33, wherein the imaging assembly comprises an image sensor and apreceding color filter array which has a tunable spectral response. 41.A non-transitory computer-readable medium having computer-executableprocess steps stored thereon for adjusting white balance of an imagecapture device which includes an imaging assembly having a spectralresponse which is tunable in accordance with a capture parameter,wherein said process steps comprise: a first applying step to apply afirst capture parameter to the imaging assembly, and capturing a firstimage of the scene using the imaging assembly whose spectral response istuned in accordance with the first capture parameter; a second applyingstep to apply a second capture parameter to the imaging assembly,wherein the second capture parameter is different from the first captureparameter, and capturing a second image of the scene by using theimaging assembly whose spectral response is tuned in accordance with thesecond capture parameter; an identifying step to identify an illuminantbased on one or more color channel ratios of the first image of thescene and one or more color channel ratio of the second image of thescene; a deriving step to derive a third capture parameter in accordancewith the identified illuminant for the scene, wherein the third captureparameter is derived so as to obtain white balance in accordance withthe identified illuminant; and a third applying step to apply the thirdcapture parameter to the imaging assembly.
 42. The non-transitorycomputer-readable medium according to claim 41, wherein said processsteps further comprise a comparing step, wherein the first and secondimages are compared to identify multiple regions of the scene having asimilar illuminant, and wherein the third capture parameter is appliedto the imaging assembly for all of such multiple regions.
 43. Thenon-transitory computer-readable medium according to claim 41, whereinsaid process steps further comprise a comparing step, wherein the firstand second images are compared to identify multiple regions in the scenehaving illuminants that differ from one another, and further comprisethe step of deriving multiple third capture parameters, one each foreach of such multiple regions, and wherein each such third captureparameter is applied to the imaging assembly in correspondence to eachof such multiple regions.
 44. The non-transitory computer-readablemedium according to claim 41, wherein the first and second captureparameters are sufficiently different from each other so as to providegood discrimination for the illuminant.
 45. The non-transitorycomputer-readable medium according to claim 41, wherein one of the firstand second capture parameters causes a blue-shift in spectralsensitivity relative to the other of the first and second captureparameters.
 46. The non-transitory computer-readable medium according toclaim 41, wherein the imaging assembly provides three or more channelsof information for each pixel of the scene, including a red-likechannel, a blue-like channel and a green-like channel, and wherein insaid identifying step, the illuminant is identified based at least on ared/green ratio and a red/blue ratio of the first image and a red/greenratio and a red/blue ratio of the second image.
 47. The non-transitorycomputer-readable medium according to claim 41, wherein the imagingassembly comprises an image sensor which has a tunable spectralresponse.
 48. The non-transitory computer-readable medium according toclaim 41, wherein the imaging assembly comprises an image sensor and apreceding color filter array which has a tunable spectral response. 49.An apparatus comprising: an imaging assembly having a spectral responsewhich is tunable in accordance with a capture parameter; a memory forstoring computer-executable process steps; and a processor for executingthe computer-executable process steps stored in the memory, wherein thecomputer-executable process steps comprise code for executing a firstapplying step to apply a first capture parameter to the imagingassembly, and capturing a first image of the scene using the imagingassembly whose spectral response is tuned in accordance with the firstcapture parameter; code for executing a second applying step to apply asecond capture parameter to the imaging assembly, wherein the secondcapture parameter is different from the first capture parameter, andcapturing a second image of the scene using the imaging assembly whosespectral response is tuned in accordance with the second captureparameter; and code for executing an identifying step to identify anilluminant for the scene based on at least one color channel ratio ofthe first image of the scene and at least one color channel ratio of thesecond image of the scene, wherein the illuminant has a defined spectralpower distribution.
 50. The apparatus according to claim 49, whereinsaid process steps further comprise a comparing step, wherein the firstand second images are compared to identify multiple regions of the scenehaving a similar illuminant.
 51. The apparatus according to claim 49,wherein said process steps further comprise a comparing step, whereinthe first and second images are compared to identify multiple regions inthe scene having illuminants that differ from one another.
 52. Theapparatus according to claim 49, wherein the first and second captureparameters are sufficiently different from each other so as to providegood discrimination for the illuminant.
 53. The apparatus according toclaim 49, wherein one of the first and second capture parameters causesa blue-shift in spectral sensitivity relative to the other of the firstand second capture parameters.
 54. The apparatus according to claim 49,wherein the imaging assembly provides three or more channels ofinformation for each pixel of the scene, including a red-like channel, ablue-like channel and a green-like channel, and wherein in saididentifying step, the illuminant is identified based at least on ared/green ratio and a red/blue ratio of the first image and a red/greenratio and a red/blue ratio of the second image.
 55. The apparatusaccording to claim 49, wherein the imaging assembly comprises an imagesensor which has a tunable spectral response.
 56. The apparatusaccording to claim 49, wherein the imaging assembly comprises an imagesensor and a preceding color filter array which has a tunable spectralresponse.
 57. An apparatus comprising: an imaging assembly having aspectral response which is tunable in accordance with a captureparameter; a memory for storing computer-executable process steps; and aprocessor for executing the computer-executable process steps stored inthe memory; wherein the computer-executable process steps comprise: codefor executing a first applying step to apply a first capture parameterto the imaging assembly, and capturing a first image of the scene usingthe imaging assembly whose spectral response is tuned in accordance withthe first capture parameter; code for executing a second applying stepto apply a second capture parameter to the imaging assembly, wherein thesecond capture parameter is different from the first capture parameter,and capturing a second image of the scene by using the imaging assemblywhose spectral response is tuned in accordance with the second captureparameter; code for executing an identifying step to identify anilluminant based on one or more color channel ratios of the first imageof the scene and one or more color channel ratios of the second image ofthe scene; code for executing a deriving step to derive a third captureparameter in accordance with the identified illuminant for the scene,wherein the third capture parameter is derived so as to obtain whitebalance in accordance with the identified illuminant; and code forexecuting a third applying step to apply the third capture parameter tothe imaging assembly.
 58. The apparatus according to claim 57, whereinsaid process steps further comprise a comparing step, wherein the firstand second images are compared to identify multiple regions of the scenehaving a similar illuminant, and wherein the third capture parameter isapplied to the imaging assembly for all of such multiple regions. 59.The apparatus according to claim 57, wherein said process steps furthercomprise a comparing step, wherein the first and second images arecompared to identify multiple regions in the scene having illuminantsthat differ from one another, and further comprise the step of derivingmultiple third capture parameters, one each for each of such multipleregions, and wherein each such third capture parameter is applied to theimaging assembly in correspondence to each of such multiple regions. 60.The apparatus according to claim 57, wherein the first and secondcapture parameters are sufficiently different from each other so as toprovide good discrimination for the illuminant.
 61. The apparatusaccording to claim 57, wherein one of the first and second captureparameters causes a blue-shift in spectral sensitivity relative to theother of the first and second capture parameters.
 62. The apparatusaccording to claim 57, wherein the imaging assembly provides three ormore channels of information for each pixel of the scene, including ared-like channel, a blue-like channel and a green-like channel, andwherein in said identifying step, the illuminant is identified based atleast on a red/green ratio and a red/blue ratio of the first image and ared/green ratio and a red/blue ratio of the second image.
 63. Theapparatus according to claim 57, wherein the imaging assembly comprisesan image sensor which has a tunable spectral response.
 64. The apparatusaccording to claim 57, wherein the imaging assembly comprises an imagesensor and a preceding color filter array which has a tunable spectralresponse.